\input texinfo
@comment %**start of header
@setfilename poke.info
@include version.texi
@settitle GNU poke Manual
@afourpaper
@defcodeindex pv
@comment %**end of header

@copying
This manual describes GNU poke (version @value{VERSION},
@value{UPDATED}).

Copyright @copyright{} 2019, 2020, 2021, 2022, 2023 The poke authors.

@quotation
You can redistribute it and/or modify this manual under the terms of
the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any
later version.
@end quotation
@end copying

@dircategory Editors
@direntry
* poke: (poke). Interactive editor for binary files.
@end direntry

@titlepage
@title GNU poke Manual
@subtitle for version @value{VERSION}, @value{UPDATED}
@author by Jose E. Marchesi et al.
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top GNU poke Manual

@insertcopying
@end ifnottex

GNU poke is an interactive, extensible editor for binary data.  Not
limited to editing basic entities such as bits and bytes, it provides
a full-fledged procedural, interactive programming language designed
to describe data structures and to operate on them.

This manual describes the @code{poke} binary data editor as well as
the Poke language.

@menu
Getting Started
* Introduction::		Introducing GNU poke.
* Setting Up::                  Preparing to use GNU poke.

Using poke
* Basic Editing::		Poking bits, bytes and simple data.
* Structuring Data::		Poking your own abstractions.
* Maps and Map-files::          Working with views of data.
* Writing Pickles::		Sharing your abstractions the sane way.
* Writing Binary Utilities::    Standalone Poke programs.
* Configuration::		Tailoring the tool to your needs.

Pickles
* IOS::                         Functions to work with IO spaces.
* Time::			Handling time.
* Colors::			Editing colors.
* Audio::			Poking audio files.
* Object Formats::		Editing executables and libraries.
* Programs::			Facilities to write binary utilities.

poked
* poked::                       The poke daemon.

poke and Emacs
* Programming Emacs Modes::     Modes to write Poke and RAS programs.

poke and Vim
* Vim Syntax Highlighting::     Poke Syntax Highlighting in Vim.

Reference Material
* Dot-Commands::                Commanding the tool.
* Commands::			Commands for editing data.
* The Poke Language::		Poke reference manual.
* The Standard Library::	Standard goodies for Poke programs.

Internals
@c * Hacking Poke::		Extending poke.
* The Poke Virtual Machine::	The PVM and its mysteries.

Appendices
* Table of ASCII Codes::	   The ASCII character set.
* GNU Free Documentation License:: Distribution terms for this document.
* Indices::			   List of indices in this document.

@detailmenu
@ifnothtml
--- The Detailed Node Listing ---
@end ifnothtml

Here are some other nodes which are really subnodes of the ones
already listed, mentioned here so you can get to them in one step:

Introduction
* Motivation::			Why a binary editor?
* Nomenclature::		poke, Poke and pickles.
* Invoking poke::		command line options.
* Commanding poke::		Interactive and non-interactive usage.

Setting Up
* Setting up Hyperlinks::       GNU poke uses terminal hyperlinks.
* Simple Init File::            A simple initial ~/.pokerc.

Basic Editing
* Binary Files::		Text vs.@: binary.
* Files as IO Spaces::		Poking files.
* Dumping File Contents::	A first look at a file's bytes.
* Poking Bytes::		Reading, manipulating and writing bytes.
* Values and Variables::        Values can be stored in variables.
* From Bytes to Integers::	Building numbers with bytes.
* Big and Little Endians::	Pick your egg.
* Negative Integers::		Going behind zero.
* Weird Integers::		Incomplete bytes in numbers.
* Unaligned Integers::		IO spaces are bit-oriented.
* Integers of Different Sizes:: Promotion of integers in expressions.
* Offsets and Sizes::           United values.
* Buffers as IO Spaces::	Poking memory buffers.
* Copying Bytes::		Moving data between IO spaces.
* Saving Buffers in Files::	From memory to files.
* Character Sets::		ASCII, Unicode, @dots{}
* From Bytes to Characters::	Working with ASCII codes
* ASCII Strings::		NULL-terminated strings.
* From Strings to Characters::	Indexing strings.

Structuring Data
* The SBM Format::		The Stupid BitMap Format
* Poking a SBM Image::		A need for abstraction.
* Modifying SBM Images::	Modifying existing data.
* Defining Types::		Abstracting data structures.
* Pickles::			Pickling useful abstractions.
* Poking Structs::		Abstracting heterogeneous data.
* How Structs are Built::	How poke builds struct values.
* Variables in Structs::
* Functions in Structs::
* Struct Methods::		Operations provided by built structs.
* Padding and Alignment::       Reserved fields, payloads and more.
* Dealing with Alternatives::   Conditional decoding.
* Structured Integers::         Structs that are stored like integers.
* Working with Incorrect Data:: Non-strict mapping to the rescue.

Maps and Map-files
* Editing using Variables::     Data is usually edited mapping variables.
* poke Maps::                   Maps and map-files.
* Loading Maps::                Using maps defined in files.
* Multiple Maps::               Multiple perspectives of the same data.
* Auto-map::                    Loading maps automagically.
* Constructing Maps::           Creating and managing maps on the fly.
* Predefined Maps::             Collection of already-written maps.

Writing Pickles
* Pretty-printers::		Conventions for pretty-printed output.
* Setters and Getters::		Anatomy getters and setters.

Writing Binary Utilities
* Poke Scripts::                Using poke as an interpreter.
* Command-Line Arguments::      Handling command-line arguments.
* Exiting from Scripts::        Exit with an error status.
* Loading pickles as Modules::  Importing pickles in programs.
* elfextractor::                An example Poke binary utility.
* Filters::                     Writing filters for binary data.

Configuration
* pokerc::			User's initialization file.
* Load Path::			Determining location of modules.
* Styling::			Changing the appearance of poke's output.

Time
* POSIX Time::			Encoding POSIX dates.

Colors
* The Color Registry::		The @file{color} pickle.
* RGB24 Encoding::		Encoding colors with three bytes.

Audio
* MP3::				Editing MP3 files.

Programming Emacs Modes
* poke-mode::                   Major mode for writing Poke programs.
* poke-map-mode::               Major mode for writing map-files.
* poke-ras-mode::               Major mode for writing RAS programs.

Vim Syntax Highlighting
* @file{poke.vim}::    Poke syntax highlighter

Dot-Commands
* load command::		Loading pickles.
* source command::              Executing commands in files.
* file command::		Opening and selecting file IO spaces.
* mem command::			Opening and selecting memory IO spaces.
* nbd command::			Opening and selecting NBD IO spaces.
* proc command::                Opening and selecting process IO spaces.
* ios command::			Switching between IO spaces.
* close command::		Closing IO spaces.
* doc command::                 Online manual.
* editor command::		Using an external editor for input.
* info command::		Getting information about open files, @i{etc}.
* set command::			Querying and setting global options.
* vm command::			Poke Virtual Machine services.
* exit command::		Exiting poke :(
* quit command::		Likewise.

Commands
* dump::			Binary dumps.
* copy::			Copying data around.
* save::			Save data into a file.
* sdiff::                       Comparing mapped values byte by byte.
* extract::			Extract contents of values to buffers.
* scrabble::			Scrabble memory chunks based on patterns.

The Poke Language
* Integers::			Whole numbers.
* Offsets::			Memory sizes and offsets.
* Strings::			NULL-terminated strings.
* Arrays::			Homogeneous collections.
* Structs::			Heterogeneous collections.
* Types::			Declaring types.
* Assignments::			Changing the value of variables.
* Compound Statements::		Sequences of statements.
* Conditionals::		Conditional statements and expressions.
* Loops::			Statements to iterate on conditions.
* Expression Statements::	Using expressions for their side-effects.
* Functions::			Procedural abstraction.
* Assembler::                   Assembler instructions with Poke operands.
* Endianness::			Byte ordering.
* Mapping::			Accessing IO spaces.
* Exception Handling::		Dealing with exceptional conditions.
* Terminal::	                Dealing with the terminal.
* Printing::			Output in Poke programs.
* Comments::			Documenting Poke programs.
* Modules::			Loading pickles from Poke programs.
* System::			Accessing the system from within Poke.
* VM::                          Tweaking the PVM in Poke programs.
* Debugging::                   Debugging Poke programs.

The Standard Library
* Standard Integral Types::	int, long and the like.
* Standard Offset Types::	off64 and the like.
* Standard Units::		b, B, Kb and the like.
* Conversion Functions::	catos, atoi, @i{etc}.
* Array Functions::             Functions which deal with arrays.
* String Functions::		Functions which deal with strings.
* Character Functions::         Functions which deal with characters.
* Values Functions::            Functions which deal with values in general.
* Sorting Functions::		qsort.
* CRC Functions::               Cyclic Redundancy Checksums.
* Dates and Times::             Processing and displaying dates and times.
* Offset Functions::            Useful functions that operate on offsets.
* Other Functions::             Miscellany.

@c Hacking poke
@c * Writing Commands::		Extending poke with new commands.

The Poke Virtual Machine
* PVM Instructions::		Virtual Machine instructions.
@end detailmenu
@end menu

@node Introduction
@chapter Introduction

@menu
* Motivation::			Why a binary editor?
* Nomenclature::		poke, Poke and pickles.
* Invoking poke::		command line options.
* Commanding poke::		Interactive and non-interactive usage.
@end menu

@node Motivation
@section Motivation

The main purpose of GNU poke is to manipulate structured binary data
in terms of abstractions provided by the user.  The Poke type
definitions can be seen as a sort of declarative specifications for
decoding and encoding procedures.  The user specifies the structure of
the data to be manipulated, and poke uses that information to
automagically decode and encode the data.  Under this perspective,
struct types correspond to sequences of instructions, array types to
repetitions or loops, union types to alternatives or conditionals, and
so on.

@menu
* Decode-Compute-Encode:: Encoded data and conventional languages.
* Describe-Compute::      The poke approach.
@end menu

@node Decode-Compute-Encode
@subsection Decode-Compute-Encode

Computing with data whose form is not the most convenient way to be
manipulated, like is often the case in unstructured binary data,
requires performing a preliminary step that transforms the data into a
more convenient representation, usually featuring a higher level of
abstraction.  This step is known in computer jargon as
@dfn{unmarshalling}, when the data is fetch from some storage or
transmission media or, more generally, @dfn{decoding}.

Once the computation has been performed, the result should be
transformed back to the low-level representation to be stored or
transmitted.  This is performed in a closing step known as
@dfn{marshalling} or, more generally, @dfn{encoding}.

Consider the following C program whose purpose is to read a 32-bit
signed integer from a byte-oriented storage media at a given offset,
multiply it by two, and store the result at the same offset.

@example
void double_number (int fd, off_t offset, int endian)
@{
   int number, i;
   unsigned char b[4];

   /* Decode.  */
   lseek (fd, offset, SEEK_SET);
   for (i = 0; i < 4; ++i)
      read (fd, &b[i], 1);

   if (endian == BIG)
     number = b[0] << 24 | b[1] << 16 | b[2] << 8 | b[3];
   else
     number = b[3] << 24 | b[2] << 16 | b[1] << 8 | b[0];

   /* Compute.  */
   number = number * 2;

   /* Encode.  */
   if (endian == BIG)
   @{
     b[0] = (number >> 24) & 0xff;
     b[1] = (number >> 16) & 0xff;
     b[2] = (number >> 8) & 0xff;
     b[3] = number & 0xff;
   @}
   else
   @{
     b[3] = (number >> 24) & 0xff;
     b[2] = (number >> 16) & 0xff;
     b[1] = (number >> 8) & 0xff;
     b[0] = number & 0xff;
   @}

   lseek (fd, offset, SEEK_SET);
   for (i = 0; i < 4; ++i)
      write (fd, &b[i], 1);
@}
@end example

As we can see, decoding takes care of fetching the data from the
storage in simple units, bytes.  Then it mounts the more abstract
entity on which the computation will be performed, in this case a
signed 32-bit integer.  Considerations like endianness, negative
encoding (which is assumed to be two's complement in this example and
handled automatically by C) and error conditions (omitted in this
example for clarity) should be handled properly.

Conversely, encoding turns the signed 32-bit integer into a sequence
of bytes and then writes them out to the storage at the desired
offset.  Again, this requires taking endianness into account and
handling error conditions.

This example may look simplistic and artificial, and it is, but too
often the computation proper (like multiplying the integer by two) is
way more straightforward than the decoding and encoding of the data
used for the computation.

Generally speaking, decoding and encoding binary data is laborious and
error prone.  Think about sequences of elements, variable-length and
clever compact encodings, elements not aligned to byte boundaries, the
always bug-prone endianness, and a long @i{etc}.  Dirty business,
sometimes risky, and always @emph{boring}.

@node Describe-Compute
@subsection Describe-Compute

This is where poke comes into play.  Basically, it allows you to
describe the characteristics of the data you want to compute on, and
then decodes and encodes it for you, taking care of the gory details.
That way you can concentrate your energy on the @emph{fun} part:
computing on the data at your pleasure.

Of course, you are still required to provide a description of the
data.  In the Poke language, these descriptions take the form of
@dfn{type definitions}, which are @dfn{declarative}: you specify
@emph{what} you want, and poke extracts the @emph{how} from that.

For example, consider the following Poke type definition:

@example
type Packet =
  struct
  @{
    uint<16> magic == 0xef;
    uint<32> size;
    byte[size] data @@ 8#B;
  @};
@end example

This tells poke that, in order to decode a @code{Packet}, it should
perform the following procedure (a similar procedure is implied for
encoding):

@itemize @minus
@item
Read two bytes from the IO space, mount them into an unsigned 16-bit
integer using whatever current endianness, and put it in
@code{magic}.  If this unsigned 16-bit integer doesn't equal to
@code{0xef}, then stop and emit a ``data integrity'' error.
@item
Read four bytes, mount them into an unsigned 32-bit integer using the
same endianness, and put it in @code{size}.
@item
Seek the IO space to advance 16 bits.
@item
Do @code{size} times:

@itemize @minus
@item
Read one byte and mount it into an unsigned 8-bit integer.
@item
Put the integer in the proper place in the @code{data} array.
@end itemize
@end itemize

If during this procedure an end of file is encountered, or some other
erroneous condition happens, an appropriate error is raised.

In the procedure sketched above we find a sequence of operations,
implied by the struct type, and a loop, implied by the array type.  As
we shall see later in this book, it is also possible to decode
conditionally.  Union types are used for that purpose.

@node Nomenclature
@section Nomenclature
@cindex poke
@cindex Poke
@cindex pickle

GNU poke is a new program and it introduces many new concepts.  It is
a good idea to clarify how we call things in the poke community.
Unless everyone uses the same nomenclature to refer to pokish
thingies, it is gonna get very confusing very soon!

First of all we have @command{poke}, the program.  Since ``poke'' is a
very common English word, when the context is not clear we either use
the full denomination @command{GNU poke}, or quote the word using some
other notation.

Then we have @dfn{Poke}, with upper case P, which is the name of the
domain-specific programming language implemented by @command{poke},
the program.

This distinction is important.  For example, when people talk about
``poke programmers'' they refer to the group of people hacking GNU
poke.  When they talk about ``Poke programmers'' they refer to the
people who write programs using the Poke programming language.

Finally, a @dfn{pickle} is a Poke source file containing definitions
of types, variables, functions, @i{etc}, that conceptually apply to some
definite domain.  For example, @file{elf.pk} is a pickle that provides
facilities to poke ELF object files.  Pickles are not necessarily
related to file formats: a set of functions to work with bit patterns,
for example, could be implemented in a pickle named
@file{bitpatterns.pk}.

We hope this helps to clarify things.

@node Invoking poke
@section Invoking poke
@cindex invoking
Synopsis:

@example
poke [@var{option}@dots{}] [@var{file}]
@end example

@noindent
The following options are available.

@table @samp
@item -l
@itemx --load=@var{file}
Load the given file as a Poke program.  Any number of @samp{-l}
options can be specified, and they are loaded in the given order.

@item -L @var{file}
Load the given file as a Poke program and exit.  The rest of the
command-line is not processed by poke, and is available to the Poke
script in the @code{argv} variable.  This is commonly used along with
a shebang (@pxref{Scripts}) to implement Poke scripts.
@end table

@noindent
Commanding poke from the command line:

@table @samp
@item -c
@itemx --command=@var{cmd}
Execute the given command.  Any number of @samp{-c} options can be
specified, and they are executed in the given order.

@item -s
@itemx --source=@var{file}
Load @var{file} as a command file.  Any number of @samp{-s} options may
be specified, and they are loaded in the given order. @xref{Command
Files}.
@end table

@noindent
Styling text output:

@table @samp
@item --color=@var{how}
@cindex styled output
Whether to use styled output, and how.  Valid options for @var{how}
are @samp{yes}, @samp{no}, @samp{auto}, @samp{html} and @samp{test}.

@item --style-dark
Use the default style that works better for dark backgrounds.  This is
the default.

@item --style-bright
Use the default style that works better for bright backgrounds.

@item --style=@var{file}
Use @var{file} as the CSS to use for styling poke, instead of the
default style.
@end table

@noindent
Other options:

@table @code
@item -q
@itemx --no-init-file
Do not load the @file{~/.pokerc} init file.
@cindex @file{.pokerc}

@item --no-auto-map
Do not load map-files automatically when poke opens IO spaces.

@item --no-hserver
Do not run the terminal hyperlinks server.

@item --quiet
Be as terse as possible.

@item --help
Print a help message and exit.

@item --version
Show version and exit.
@end table

@node Commanding poke
@section Commanding poke

GNU poke is primarily an interactive editor that works in the command
line.  However, it is also possible to use it in a non-interactive
way.  This chapter documents both possibilities.

@menu
* The REPL::                  Using poke interactively.
* Evaluation::		      Evaluating Poke.
* Commands and Dot-Commands:: Two kinds of commands.
* Command Files::	      Loading commands from files.
* Scripts::		      Executing Poke programs in the command line.
@end menu

@node The REPL
@subsection The REPL
@cindex REPL
If poke is invoked with an interactive TTY connected to the standard
input, it greets you with a welcome message, licensing
information and such, and finally a prompt that looks like:

@example
(poke)
@end example

At this point, the program is ready to be commanded.  You are
expected to introduce a line and press @key{enter}.  At that point
poke will examine the command, notify you if there is some error
condition, process the line and maybe displaying something in the
terminal.

@cindex readline
Repeatedly typing complex commands can be tiresome.
To help you, poke uses the readline library
@xref{Top,,,rluserman,GNU Readline Library}.
This provides shortcuts and simple keystrokes to repeat
previous commands with or without modification, fast selection of
file names and entries from other multiple choice contexts, and
navigation within a command and among previous commands.
@cindex history, session history
When the REPL starts, the history of your previous sessions
are loaded from the file @file{.poke_history} located in your home
directory (if it exists).

There are several kinds of lines that can be provided in the REPL:

@itemize @bullet
@item A @dfn{dot-command} invocation, that starts with a dot character
(@command{.}).
@item A command invocation.
@item A Poke statement.
@item A Poke expression.
@end itemize

These are explained in the following sections.

@node Evaluation
@subsection Evaluation

You can evaluate a Poke statement by typing it at the REPL's prompt.
Only a single statement, including expression statements
(@pxref{Expression Statements}) and compound statements
(@pxref{Compound Statements}), can be evaluated this way.  It needs not
be terminated by a semicolon.

When an expression is evaluated, the result of the evaluation is
printed back to you.  For example:

@cindex expressions
@example
(poke) 23
23
(poke) [1,2,3]
[1,2,3]
(poke) Packet @@ 0#B
Packet @{i=1179403647,j=65794L@}
@end example

@cindex statements
When a statement other than an expression statement is executed in the
REPL no result is printed, but of course the statement can print on its
own:

@example
(poke) fun do_foo = void: @{@}
(poke) do_foo
(poke) for (i in [1,2,3]) printf "elem %i32d\n", i;
elem 1
elem 2
elem 3
@end example

@cindex errors
If there is an error compiling the line, you are notified with a
nice error message, showing the location of the error.  For example:

@example
(poke) [1,2,3 + "foo"]
<stdin>:1:6: error: invalid operands in expression
[1,2,3 + "foo"];
     ^~~~~~~~~
@end example

@node Commands and Dot-Commands
@subsection Commands and Dot-Commands

There are two kinds of commands in poke: the @dfn{dot-commands}, which
are written in C and have their own conventions for handling
sub-commands and passing arguments and flags, and normal commands,
which are written in Poke.

@subsubsection Dot-Commands
@cindex dot-commands
Dot-commands are so called because their names start with the dot
character (@code{.}).  They can feature subcommands.  Example:

@example
(poke) .vm disassemble expression int[] @@ 0#B
(poke) .vm disassemble function (dump)
@end example

When there is no ambiguity, the command name and the subcommands can
be shortened to prefixes.  The commands above can also be written as:

@example
(poke) .vm dis e int[] @@ 0#B
(poke) .vm dis f (dump)
@end example

@cindex flags
Some commands also get flags, which are one-letter indicators that can
be appended to the command name (including subcommands) after a slash
character (@code{/}).  For example, the @command{.vm} disassembler
commands accept a @code{n} flag to indicate we want a native
disassemble.  We can pass it as follows:

@example
(poke) .vm disassemble expression/n int[] @@ 0#B
(poke) .vm disassemble function/n (dump)
@end example

If a dot-command accepts more than one argument, they are separated
using comma characters (@code{,}).  Spaces are generally ignored.

@subsubsection Commands
@cindex commands
Regular poke commands are written in Poke and use different
conventions.  The name of commands follow the same rules as normal
Poke identifiers, and do not start with a dot character.

An example is the @command{dump} command:

@example
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0100 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 8001 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@@.....@@.....
00000040: 1800 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 7900 0000 0000 0000 b701 0000 9a02 0000  y...............
00000060: 7b10 0000 0000 0000 1800 0000 0000 0000  @{...............
00000070: 0000 0000 0000 0000 7900 0000 0000 0000  ........y.......
@end example

@cindex arguments
After the name of the command, arguments can be specified by name,
like this:

@example
(poke) dump :from 0#B :size 8#B
(poke) dump :from 0#B :size 8#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100                      .ELF....
@end example

The @command{dump} command is discussed in greater detail
below (@pxref{dump}).
The order of arguments is irrelevant in principle:

@example
(poke) dump :from 0#B :size 8#B :ascii 0 :ruler 0
00000000: 7f45 4c46 0201 0100
(poke) dump :ruler 0 :from 0#B :size 8#B :ascii 0
00000000: 7f45 4c46 0201 0100
@end example

However, beware side effects while computing the values you pass as
the arguments!  The expressions themselves are evaluated from left to
right (according to the order of declaration in the signature).

Which arguments are accepted, and their kind, depend on the specific
command.

Note that the idea is to restrict the number of dot-commands to the
absolutely minimum.  Most of the command-like functionality provided
in poke shall be implemented as regular commands.

@node Command Files
@subsection Command Files
@cindex command files
Command files contain poke commands.
A poke command may be a dot command, a Poke statement or a Poke
expression.
Lines starting with @code{#} are comments will be ignored.   However a
comment must start at the beginning of a line.  Here is an example
of a script:

@example
# The following two lines are dot commands
.load my-pickle.pk
.set obase 16

# The following line is a Poke statement
dump :size 0x100#B :from 0x10#B

# The following line is a Poke expression statement without any side effect.
# Consequently it is valid, but rather useless.
4 == 4
@end example

A command file contains commands, not Poke code.  This means it gets
read line by line and commands cannot occupy more than one line.
Hence the following is a valid command file:

@example
type foo = struct @{int this; int that;@}
@end example

@noindent
but this is not valid as a command file (although it is a valid Poke
statement) and will provoke an error:

@example
type foo = struct
@{
 int this;
 int that;
@}
@end example

Command files can be loaded at startup using the @code{-s} command
line option (@pxref{Invoking poke}).
The @file{~/.pokerc} startup file is also an example of a poke
command file (@pxref{pokerc}).

@node Scripts
@subsection Scripts
@cindex shebang, @code{#!}
@cindex scripts
Following the example of Guile Scheme, the Poke syntax includes
support for multi-line comments using the @code{#!} and @code{!#}
delimiters.  This, along with the @code{-L} command line option,
allows to write Poke scripts and execute them in the command line like
if they were normal programs.  Example of a script:

@example
#!/usr/bin/poke -L
!#

print "Hello world!\n";
@end example

The resulting script can process command-line options by accessing the
@code{argv} array.  The following Poke script prints its arguments:

@example
#!/usr/bin/poke -L
!#

for (arg in argv)
  printf ("Argument: %s\n", arg);
@end example

If you want to pass additional flags to the poke command, you need to
use a slightly different kind of shebang:

@example
#!/usr/bin/env sh
exec poke -L "$0" "$@@"
!#

load elf;
open (argv[0]);
printf ("%v\n", Elf64_Ehdr @@ 0#B);
@end example

@node Setting Up
@chapter Setting Up

@menu
* Setting up Hyperlinks::       GNU poke uses terminal hyperlinks.
* Simple Init File::            A simple initial ~/.pokerc.
@end menu

@node Setting up Hyperlinks
@section Setting up Hyperlinks

GNU poke uses @dfn{terminal hyperlinks} in order to improve the
interactive usage of the tool: clicking on terminal hyperlinks
requests poke to execute certain actions.  This is used to implement
buttons and other interactive goodies.

Many terminal emulators support terminal hyperlinks.  However, we are
using a very simple protocol called @code{app://} that is not
supported (yet) on GNU/Linux distros.  Fortunately, it is very easy to
set your system to use this protocol, and this chapter shows you how.

@subsection Make sure your poke speaks hyperlinks

The first step in having an hyperlinks-capable poke is to make sure to
have a recent enough version of libtextstyle when building poke.  If
your poke can emit hyperlinks you will see a message like this when
running it on the terminal:

@example
hserver listening in port 43713.
@end example

@subsection Use a terminal emulator that supports hyperlinks

Gnome Terminal has support for displaying hyperlinks as do many other
emulators that rely on VTE. Check the list at
@url{https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#supporting-apps}
for a mostly up-to-date, non-exhaustive list of emulators that support
printing hyperlinks.

@subsection Get and install the app-client utility

Since @code{app://} is a new URI protocol that we designed, common
terminal emulators don't know what to do when they encounter such a
URI. To work around this problem we use the XDG Desktop Specification
and a little C utility called @command{app-client}, which can be found
at @url{https://gitlab.com/darnir/hyperlink-app-client}.

By setting @command{app-client} as the default handler for @code{app://}
URIs, the terminal emulator does not need to understand the syntax or
semantics of the @code{app://} protocol. It offloads the handling of
the URI entirely to @command{app-client}. In order to use this, first download
and install @command{app-client}:

@example
$ git clone https://gitlab.com/darnir/hyperlink-app-client
$ cd hyperlink-app-client
$ make
$ sudo make install
@end example

This is enough for any utility (like terminals) that use
@command{xdg-open} to do the right thing with hyperlinks.  However,
certain terminals require additional setup.  See below if that is your
case.

@subsection GNOME Terminal

Gnome Terminal doesn't use @command{xdg-open} to start the
applications. Instead, it parses the @file{mimeapps.list} file
manually to find the right application.

Edit your @file{mimeapps.list}, it is usually located at
@file{$HOME/.local/share/applications/mimeapps.list}, but it might
also be at @file{$XDG_CONFIG_DIR/mimeapps.list}, and add the following
line to it:

@example
x-scheme-handler/app=app-client.desktop
@end example

This let's Gnome Terminal know how to open @code{app://} links.

@node Simple Init File
@section Simple Init File

GNU poke is a spartan program that tries to be as simple as possible
by default, without fancy displays.  Therefore, before exploring poke
you may want to configure it minimally.  This section contains a few
recommendations in that respect.

First we must say that poke reads a per-user configuration from the
@file{~/.pokerc} file.  @xref{pokerc}.

We recommend new users to set the following options:

@example
.set endian little
.set omode tree
.set oacutoff 5
.set pretty-print yes
.set pager yes
@end example

@noindent
@code{.set endian little} will make poke to use little-endian by
default when accessing IO spaces, such as files or memory buffers.
@xref{Big and Little Endians}.

@code{.set omode tree} will make poke to format composite data
structures, such as arrays and struct, in several lines in a nice
tree-like format.  By default the program will format these values in
a single line (what we call the ``plain'' output mode) which can be
difficult to read.

@code{.set oacutoff 5} will make poke to output at most five elements
in array values, followed by a dieresis @code{...}.

@code{.set pretty-print yes} will make poke to use pretty-printers
when printing out values of struct types that have a pretty-printer
defined for them.  You can recognize pretty-printed values because, by
convention, they are emitted between @code{#<} and @code{>} markers,
like for example @code{#<leb128:1234>}.  If you see a pretty-printed
value and you need to look at its internal structure, you can disable
pretty-printing using a @code{.set pretty-print no} dot-command at any
time. @xref{Pretty-printers}.

@code{.set pager yes} will make poke to page the output of commands
that emit output larger than the visible terminal.  This of course
only has an effect when running poke interactively.

These options are fully explained later in this manual.  @xref{set
command}.

@node Basic Editing
@chapter Basic Editing

In this chapter you will learn how to shuffle binary data around with
poke, in terms of fundamental predefined entities: bits, bytes,
integers, and the like.

@menu
* Binary Files::		Text vs.@: binary.
* Files as IO Spaces::		Poking files.
* Dumping File Contents::	A first look at a file's bytes.
* Poking Bytes::		Reading, manipulating and writing bytes.
* Values and Variables::        Values can be stored in variables.
* From Bytes to Integers::	Building numbers with bytes.
* Big and Little Endians::	Pick your egg.
* Negative Integers::		Going behind 0.
* Weird Integers::		Incomplete bytes in numbers.
* Unaligned Integers::		IO spaces are bit-oriented.
* Integers of Different Sizes:: Promotion of integers in expressions.
* Offsets and Sizes::           United values.
* Buffers as IO Spaces::	Poking memory buffers.
* Copying Bytes::		Moving data between IO spaces.
* Saving Buffers in Files::	From memory to files.
* Character Sets::		ASCII, Unicode, @dots{}
* From Bytes to Characters::	Working with ASCII codes
* ASCII Strings::		NULL-terminated strings.
* From Strings to Characters::	Indexing strings.
* Strings are not Arrays::      Converting between strings and arrays.
@end menu

@node Binary Files
@section Binary Files

GNU poke is an editor for @dfn{binary files}.  Right, so what is a
binary file?  Strictly speaking, every file in a computer's file
system is binary.  This is because, in a very fundamental level,
files are just sequences of bytes.

@cindex binary files
@cindex text files
Colloquially, however, it is very common to talk about ``binary
files'' as opposed to ``text files''.  In this informal meaning, a
text file is basically a file composed, mostly, of bytes (and byte
sequences) that can be translated into printable characters in some
character set, such as ASCII, EBCDIC or Unicode.  It follows that
binary files would then be files composed, mostly, of bytes not
intended to be interpreted as encoded characters.

Some text files contain non-printable characters, such as form feed
characters, and many binary files contain printable strings, such as a
string table in an ELF object file.  That is why we used the word
``mostly'' in the definitions above.  In practice, however, the
distinction is almost always clear and there is common consensus on
whether a given file format can be considered as a binary format, or
not.

GNU poke can edit any file, and as we shall see, it provides some nice
features to manipulate sequences of bytes interpreted as character
strings.  However, it is called a ``binary editor'' because it is
especially designed to be particularly useful editing binary files, in
the sense of the term defined above.

In this chapter, we will be using ELF object files as the experiment
subject in most of the examples.  ELF files are good for this purpose,
because they are eminently binary, highly structured, and still
strings play a role in them, encoding names of entities like sections
and symbols.  You don't need to have a perfect knowledge of the ELF
format in order to follow the examples, but being familiarized with
the concept of object file formats should surely help.

Obtaining a simple ELF object file is easy, if you have a C compiler
installed:

@example
$ echo 'int foo () @{ return 0; @}' | gcc -c -xc -o foo.o -
@end example

The command above compiles a very simple ELF object file that contains
the compiled form of a little dummy function.  This object file will
be our companion for a while, and will be the subject of much analysis
and abuse, as we poke it.

@node Files as IO Spaces
@section Files as IO Spaces

Now that we have a binary file (@file{foo.o}) it is time to open it
with poke.  There are two ways to do that.

One way is to pass the name of the file in the poke invocation.  The
program will start, open the file, and present you with the REPL, like
in:

@example
$ poke foo.o
[@dots{}]
(poke)
@end example

The other way is to fire up poke without arguments, and then use the
@command{.file} dot-command to open the file:

@example
$ poke
[@dots{}]
(poke) .file foo.o
The current IOS is now `./foo.o'.
(poke)
@end example

@noindent
Note how poke replies to the dot-command, stating that the
@dfn{current IOS} is now the file we opened.

You may be wondering, what is this IOS thing?  It is an acronym for
Input/Output Space, often written IO Space. This is the denomination
used to refer to the entities being edited with poke.  In this case
the IO space being edited is a file, but we will see that is not
always the case: poke can also edit other entities such as memory
buffers and remote block-oriented devices over the network.  For now,
let's keep in mind that IOS, or IO space, refers to the file being
edited.

And why ``current''?  GNU poke is capable of editing several files
(more generally, several IO spaces) simultaneously.  At any time, one
of these files is the ``current one''.  In our case, the current IO
space is the file @file{foo.o}, since it is the only file poke knows
about:

@example
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o
@end example

The command @command{.info ios} gives us information about all the IO
spaces that are currently open.  The first column tells us a @dfn{tag}
that identifies the IOS.  In this example, the tag corresponding to
@file{foo.o} is @code{#0}.  The second column tells us the type of
IO space. The third column tells us that @file{foo.o} allows both
reading and writing.  The fourth column tells us the size of the file,
in hexadecimal.

You may wonder what is that weird suffix @code{#B}.  It is a unit,
and tells us that the size @code{0x398} is measured in bytes, @i{i.e.} the
size of @file{foo.o} is @code{0x398} bytes (or, in decimal, @code{920}
bytes.)

Finally, the asterisk character at the left of the entry for
@file{foo.o} identifies it as the current IO space.  To see this more
clearly, let's open another file:

@example
(poke) .file bar.o
The current IOS is now `./bar.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	FILE	rw	0x00000000#B	0x00000398#B	./bar.o
  #0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o
@end example

Ah, there we have both @file{foo.o} and @file{bar.o}.  Now the current
IO space (the entry featuring the asterisk at the left) is the file
that we just opened, @file{bar.o}.  This is because poke always sets
the most recently open file as the current one.  We can switch back to
@file{foo.o} using yet another dot-command, @command{.ios}, which gets
an IO space tag as an argument:

@example
(poke) .ios #0
The current IOS is now `./foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
  #1	FILE	rw	0x00000000#B	0x00000398#B	./bar.o
* #0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o
@end example

@noindent
We are back to @file{foo.o}.  Since we are not really interested in
@file{bar.o}, let's close it:

@example
(poke) .close #1
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o
@end example

@noindent
Awesome.  Now we can focus on @file{foo.o}'s contents@dots{}

@node Dumping File Contents
@section Dumping File Contents

Data stored in modern computers, in both volatile memory and
persistent files, is fundamentally a sequence of entities called
@dfn{bytes}.  The bytes can be addressed by its position in the
sequence, starting with zero:

@example
+--------+--------+--------+ ... +--------+
| byte 0 | byte 1 | byte 2 |     | byte N |
+--------+--------+--------+ ... +--------+
@end example

@noindent
Each byte has capacity to store a little unsigned integer in the range
@code{0..255}.  Therefore, the IO spaces that we edit with poke (like
the file @file{foo.o}) can be seen as a sequence of little numbers,
like depicted in the figure above.

GNU poke provides a command whose purpose is to display the values of
these bytes: @command{dump}@footnote{Note that this is not a
dot-command like @command{.file}, @command{.ios} or @command{.close}:
@command{dump} does not start with a dot!  We will see later how
dot-commands differ from ``normal commands'' like @command{dump}, but
for now, let's ignore the distinction.}  .  It is called like that
because it dumps ranges of bytes to the terminal, allowing the user to
inspect them.

So let's use our first poke command!  Fire up poke, open the file
@file{foo.o} as explained above, and execute the @command{dump}
command:

@example
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@.....@.....
00000040: 2564 0a00 0000 0000 0000 0000 0000 0000  %d..............
00000050: b702 0000 0100 0000 1801 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 8510 0000 ffff ffff  ................
00000070: b700 0000 0000 0000 9500 0000 0000 0000  ................
(poke)
@end example

@noindent
What are we looking at?

The first line of the output, starting with @code{76543210}, is a
@dfn{ruler}.  It is there to help us to visually determine the
location (or offset) of the data.

The rest of the lines show the values of the bytes that are stored in
the file, 16 bytes per line.  The first column in these data lines
shows the offset, in hexadecimal and measured in number of bytes, from
which the row of data starts.  For example, the offset of the first
byte shown in the third data line has offset 0x20 in the file, the
second byte has offset 0x21, and so on.  Note how the data rows show
the values of the individual bytes, in hexadecimal.  Generally
speaking, when dealing with bytes (and binary data in general) it is
useful to manipulate magnitudes in hexadecimal, or octal.  This is
because it is easy to group digits in these bases to little groups of
bits (four and three respectively) in the equivalent binary
representation.  In this case, each couple of hexadecimal digits
denote the value of a single byte@footnote{Do not be fooled by the
fact @command{dump} shows the hexadecimal digits in groups of four:
this is just a visual aid and, as we shall see, it is possible to
change the grouping by passing arguments to @command{dump}.}.  For
example, the value of the first byte in the third data row is 0x01,
the value of the second byte 0x02, and so on.

Using the ruler and the column of offsets, locating bytes in the data
is very easy.  Let's say for example we are interested in the byte at
offset 0x68: we use the first column to quickly find the row starting
at 0x60, and the ruler to find the column marked with @code{88}.
Cross column and row and@dots{} voila!  The byte in question has the value
0x85.  The reverse process is just as easy.  What is the offset of the
first 0x40 in the file?  Try it!

The section at the right of the output is the ASCII output.  It shows
the row of bytes at the left interpreted as ASCII characters.
Non-printable characters are shown as @code{.} to avoid scrambling the
terminal, and yes, there is actually way to customize what character
to use, so they are not confused from real ASCII dot characters
(@code{0x2e}) :P In this particular dump we can see that near the
beginning of the file there are three bytes whose value, if
interpreted as ASCII characters, conform the string ``ELF''.  As we
shall see, this is part of the ELF magic number.  Again, the ruler is
very useful to locate the byte corresponding to some character in the
ASCII section, or the other way around.  What is the value of the byte
corresponding to the @code{F} in @code{ELF}?  Try it!

@cindex commands, passing arguments
Something to notice in the @command{dump} output above is that these
are not, by any mean, the complete contents of the file @file{foo.o}.
The @command{.info ios} dot-command informed us in the last section
that @file{foo.o} contains 920 bytes, of which the @command{dump}
command only showed us@dots{} @code{0x80} bytes, or @code{128} bytes in
decimal.

@command{dump} is certainly capable of showing more (and less) than
@code{128} bytes.  We can ask @command{dump} to display some given
amount of data by specifying its size using a @dfn{command argument}.
For example:

@example
(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@.....@.....
@end example

The command above asks poke to ``dump 64 bytes''.  In this example
@code{:size} is the name of the argument, and @code{64#B} is the
argument's value.  Again, the suffix @code{#B} tells poke we want to
dump 64 bytes, not 64 kilobits nor 64 potatoes.

Another interesting aspect of our first dump (ahem) is that the dumped
bytes start from the beginning of the file, @i{i.e.} the offset of the
first byte is 0x0.  Certainly there should be other areas of the file
with interesting contents for us to inspect.  To that purpose, we can
use yet another option, @code{:from}:

@example
(poke) dump :size 64#B :from 128#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000080: 1400 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000a0: 0000 0000 0300 0100 0000 0000 0000 0000  ................
000000b0: 0000 0000 0000 0000 0000 0000 0300 0300  ................
@end example

The command above asks poke to ``dump 64 bytes starting at 128 bytes
from the beginning of the file''.  Note how the first row of bytes
start at offset @code{0x80}, @i{i.e.} @code{128} in decimal.

@cindex commands, customizing
Passing options to commands is easy and natural, but we may find
ourselves passing the same values again and again to certain command
options.  For example, if the default size of @command{dump} of 128
bytes is not what you prefer, because you have a particularly tall
monitor, or you are one of these people using sub-atomic sized fonts,
it can be tiresome and error-prone to pass @command{:size} to
@command{dump} every time you use it.  Fortunately, the default size
can be customized by setting a @dfn{global variable}:

@example
(poke) pk_dump_size = 160#B
@end example

@noindent
This tells poke to set @code{160} bytes as the new value for the
@code{pk_dump_size} variable.  This is a global variable that the
@command{dump} command uses to determine how much data to show if the
user doesn't specify an explicit value with the @code{:size} option.
Many other commands use the same strategy in order to alter their
default behavior, not just @command{dump}.

@cindex .pokerc
@cindex initialization file
And now that we are talking about that, it is also cumbersome to have
to set the default size used by @command{dump} every time we run
poke.  But no problem, just set the variable in a file called
@file{.pokerc} in your home directory, like this:

@example
pk_dump_size = 160#B
@end example

@noindent
Every time poke starts, it reads @file{~/.pokerc} and executes the
commands contained in it.  See @ref{pokerc}.

The @command{dump} command is very flexible, and accepts a lot of
options and customization variables that we won't be covering in this
chapter.  For a complete description of the command, see @ref{dump}.

@node Poking Bytes
@section Poking Bytes

Let's look again at the first bytes of the file @file{foo.o}:

@example
(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@.....@.....
@end example

At this point we know how to use the ruler to localize specific bytes
just by looking at the displayed data.  If we wanted to operate on the
values of some given bytes, we could look at the dump and type the
values in the REPL.  For example, if we wanted to add the values of
the bytes at offsets 0x2 and 0x4, we could look at the dump and then
type:

@example
(poke) 0x4c + 0x02
0x4e
@end example

GNU poke supports many operators that take integers as arguments, to
perform arithmetic, relational, logical and bit-wise operations on
them (@pxref{Integers}).  Since bytes are no more (and no less) than
little unsigned integers, we can use these operators to perform
calculations on bytes.

For example, this is how we would calculate whether the highest bit in
the second byte in @file{foo.o} is set:

@example
(poke) 0x45 & 0x80
0
@end example

@noindent
Note how booleans are encoded in Poke as integers, 0 meaning false,
any other value meaning true.

Looking at the output of @command{dump} and writing the desired byte
value in the prompt is cumbersome.  Fortunately, there is a much more
convenient way to access the value of a byte, given its offset in the
file: it is called @dfn{mapping} a byte value.  This operation is
implemented by a binary operator, called the map operator.

This is how it works.  Assuming we were interested in the byte at
@code{64} bytes from the beginning of the file, this is how we would
refer to it (or ``map'' it):

@example
(poke) byte @@ 64#B
37UB
@end example

@noindent
This application of the map operator tells poke to map a byte at the
offset 64 bytes.  It can be read as ``byte at 64 bytes''.  Note how
poke replies with the value 37UB.  The suffix @code{UB} means
``unsigned byte'', and is an indication for the user about the nature
of the preceding number: it is unsigned, and it occupies a byte when
stored.

As we can see in this example, poke uses decimal by default when
showing values in the REPL.  We already noted how it is usually better
to work in hexadecimal when dealing with byte values.  Fortunately, we
can change the numeration base used by poke when printing numbers,
using the @command{.set obase} (``set output base'') dot-command as
this:

@example
(poke) .set obase 16
@end example

@noindent
After this, we can map the byte again, this time getting the result
expressed in hexadecimal:

@example
(poke) byte @@ 64#B
0x25UB
@end example

@noindent
Again, you may find it useful to add the @code{.set obase 16} command
to your @file{.pokerc} file, if you want the customization to be
persistent between poke invocations.

Going back to the example of calculating whether the highest bit in
the second byte in @file{foo.o} is set, this is how we would do it
with a map:

@example
(poke) (byte @@ 2#B) & 0x80
0
@end example

@noindent
Turns out the answer is no.

The map operator can also be used at the left side of an assignment
operator:

@example
(poke) byte @@ 0x28#B = 0xff
@end example

@noindent
Which reads ``assign 0xff to the byte at offset 0x4a bytes''.  Dumping
again, we can verify that the byte actually changed:

@example
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 ff01 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@.....@.....
@end example

@noindent
Does this mean that @file{foo.o} changed accordingly, in disk?  The
answer is yes.  poke always commits changes immediately to the file
being edited.  This, that is an useful feature, can also be a bit
tricky if you forget about it, leading to data corruption, so please
be careful.

Incidentally, altering the byte at offset 0x28 most probably have
caused @file{foo.o} to stop being a valid ELF file, but since we are
just editing bytes (and not ELF structures) we actually don't care
much.

@node Values and Variables
@section Values and Variables

Up to now we have worked with byte values, either writing them in the
REPL or mapping them at the current IO space.  Often it is useful to
save values under meaningful names, and access to them by name.  In
poke we do that by storing the values in @dfn{variables}.

Before being used, variables shall be defined using the @code{var}
construction.  Let's get the byte at offset @code{64} bytes and save it
in a variable called @code{foo}:

@example
(poke) var foo = byte @@ 64#B
@end example

@noindent
This defines a new variable (@code{foo}) and initializes it to the
value of the byte at offset 64 bytes.  This results on @code{foo} to
hold the value 37.

Once defined, we can get the value of a variable by just giving its
name to poke:

@example
(poke) foo
37UB
@end example

@noindent
Several variables can be defined in a single @code{var} declaration.
Each definition is separated by a comma.  This is equivalent of
issuing several @code{var}s:

@example
(poke) var a = 10, b = 20
@end example

@noindent
In general, a variable containing a byte value can be used in any
context where the contained value would be expected.  If we wanted to
check the highest bit in the byte value stored in @code{foo} we would
do:

@example
(poke) foo & 0x80
0x0
@end example

Assigning a value to a variable makes the value the new contents of
the variable.  For example, we can increase the value of @code{foo} by
one like this:

@example
(poke) foo = foo + 1
@end example

At this point, an important question is: when we change the value of
the variable @code{foo}, are we also changing the value of the byte
stored in @file{foo.o} at offset 64 bytes?  The answer is no.  This is
because when we do the mapping:

@example
(poke) var foo = byte @@ 64#B
@end example

@noindent
The value stored in @code{foo} is a @emph{copy} of the value returned
by the map operator @code{@@}.  You can imagine the variable as a
storage cell located somewhere in poke's memory.  After the assignment
above is executed there are two copies of the byte value 0x25: one in
@file{foo.o} at offset 64 bytes, and the other in the variable
@code{foo}.

It follows that if we wanted to increase the byte in the file, we
would need to do something like:

@example
(poke) var foo = byte @@ 64#B
(poke) foo = foo + 1
(poke) byte @@ 64#B = foo
@end example

@noindent
Or, more succinctly, omitting the usage of a variable:

@example
(poke) byte @@ 64#B = (byte @@ 64#B) + 1
@end example

@noindent
Or, even @emph{more} succinctly:

@example
(poke) (byte @@ 64#B) += 1
@end example

@noindent
Note how we have to use parenthesis around the map at the right hand
side, because the map operator @code{@@} has less precedence than the
plus operator @code{+}.

@node From Bytes to Integers
@section From Bytes to Integers

The bytes we have been working with are unsigned whole numbers (or
integers) in the range @code{0..255}.  We saw how poke sees the
contents of the files as a sequence of bytes, and how each byte can be
addressed using an offset.  Mapping bytes using the map operator
@code{@@} gives us these values, which are denoted in poke with
literals like @code{10UB} or @code{0x0aUB}.

This very limited range of values have consequences when it comes to
do arithmetic with bytes.  Suppose for example we wanted to calculate
the average of the first three byte values stored in @file{foo.o}.  We
could do something like:

@example
(poke) a0 = byte @@ 0#B
(poke) a1 = byte @@ 1#B
(poke) a2 = byte @@ 2#B
(poke) a0
0x7fUB
(poke) a1
0x45UB
(poke) a2
0x4cUB
(poke) (a0 + a1 + a2) / 3UB
5UB
@end example

@noindent
That is obviously the wrong answer.  What happened?  Let's do it step
by step.  First, we add the first two bytes:

@example
(poke) a0 + a1
0xc4UB
@end example

@noindent
Which is all right.  0xc4 is 0x7f plus 0x45.  But, let's add now the
third byte:

@example
(poke) a0 + a1 + a2
0x10UB
@end example

@noindent
That's no good.  Adding the value of the third byte (0x4c) we overflow
the range of valid values for a byte value.  The calculation went
banana at this point.

Another obvious problem is that we surely will want to store integers
bigger than 255 in our files.  Clearly we need a way to encode them
somehow, and since all we have in a file are bytes, the integers will
have to be composed of them.

Integers bigger than 255 can be encoded by interpreting consecutive
byte values in a certain way.  First, let's consider a single byte.
If we print a byte value using binary rather than decimal or
hexadecimal, we will observe that eight bits are what it takes to
encode the numbers between 0 and 0xff (255) using a @dfn{natural
binary encoding}:

@example
(poke) .set obase 2
(poke) 0UB
0b00000000UB
(poke) 0xFFUB
0b11111111UB
@end example

This is the reason why people say bytes are ``composed'' of eight
bits, or that the width of a byte is eight bits.  But this way of
talking doesn't really reflect the view that the operating system has
of devices like files or memory buffers: both disk and memory
controllers provide and consume bytes, @i{i.e.} little unsigned numbers in
the range @code{0..255}.  At that level, bytes are indivisible.  We
will see later that poke provides ways to work on the ``sub-byte''
level, but that is just really an artifact to make our life easier:
underneath, all that goes in and out are bytes.

Anyhow, if we were to ``concatenate'' the binary representation of two
consecutive bytes, we would end with a much bigger range of possible
numbers, in the range
@code{0b00000000_00000000..0b11111111_11111111}@footnote{poke allows
to insert underscore characters @code{_} anywhere in number literals.
The only purpose of these characters is to improve readability, and
they are totally ignored by poke, @i{i.e.} they do not alter the value of
the number.}, or @code{0x0000..0xffff} in hexadecimal.  poke provides
a bit-concatenation operator @code{:::} that does exactly that:

@example
(poke) 0x1UB
0b00000001UB
(poke) 0x1UB ::: 0x1UB
0b0000000100000001UH
@end example

@noindent
Note how the suffix of the resulting number is now @code{UH}.  This
indicates that the number is no longer a byte value: it is too big for
that.  The @code{H} in this new suffix means ``half'', and it is a
traditional way to call an integer that is encoded using two bytes, or
16 bits.

So, using our method of encoding bigger numbers concatenating bytes,
what would be the ``half'' integer composed of two bytes at the
beginning of @file{foo.o}?

@example
(poke) .set obase 16
(poke) (byte @@ 0#B):::(byte @@ 1#B)
0x7f45UH
@end example

Now, let's go back to the syntax we used to map a byte value.  In the
invocation of the map operator @code{byte @@ 0#B} the operand at the
left (in this case @code{byte}) tells the operator what kind of value
to map.  This is called a @dfn{type specifier}; @code{byte} is the
type specifier for a single byte value, and @code{byte[3]} is the type
specifier for a group of three byte values arranged in an array.

As it happens, @code{byte} is a synonym for another slightly more
interesting type specifier: @code{uint<8>}.  You can probably infer
the meaning already: a byte is an unsigned integer which is 8 bits
big.  We can of course use this alternate specifier in a mapping
operation, achieving exactly the same result than if we were using
@code{byte}:

@example
(poke) uint<8> @@ 0#B
0x7fUB
@end example

You may be wondering: is it possible to use a similar type specifier
for mapping bigger integers, like these ``halves'' that are composed of
two bytes?  Yeah, it is indeed possible:

@example
(poke) uint<16> @@ 0#B
0x7f45UH
@end example

@noindent
Mapping an unsigned integer of 16-bits at the offset 0 gives us an
unsigned ``half'' value, as expected.

You can easily build bigger and bigger numbers concatenating more and
more bytes.  Three bytes? sure:

@example
(poke) uint<24> @@ 0#B
(uint<24>) 0x7f454c
@end example

@noindent
Note that in this case poke uses a prefix instead of a suffix to
indicate that the given value is 24-bits long.  This is because only a
limited number of suffixes (which are more concise and more readable
than the prefix form) are available, corresponding to common or
typical widths.

Four bytes?

@example
(poke) uint<32> @@ 0#B
0x7f454c46U
@end example

Certain integer widths are so often used that easier-to-type synonyms
for their type specifiers are provided.  We already know @code{byte}
for @code{uint<8>}.  Similarly, @code{ushort} is a synonym for
@code{uint<16>}, @code{uint} is a synonym for @code{uint<32>} and
@code{ulong} is a synonym for @code{uint<64>}.  Try them!

GNU poke supports integers up to eight bytes, @i{i.e.} up to 64-bits.
This may change in the future, as we are planning to support
arbitrarily large integers.

@node Big and Little Endians
@section Big and Little Endians

@cindex big endian
@cindex little endian
@cindex endianness
When talking about whole numbers (integers) we should distinguish
between their value (such as 123) and their @dfn{written form} that we
would use when writing the number on a piece of paper, such as
@code{123}.

The written form of a number is composed of digits, arranged in
certain order.  We all know that the ordering of the digits in the
written form of a number is important: if we write @code{123} we are
referring to a different value than if we write @code{321}.  The
mathematical reason for this is that depending on the position they
occupy in the written form, each digit contributes with a different
``weight'' to the total value of the number.  This is always the case,
regardless of the numerical base used to denote the number.

For example, the value of the number 123 (whose written form is
@code{123}) is calculated as @code{1*10^2+2*10^1+3*10^0}.  If we swap
the last two digits in the written form of the number, @code{132}, we
have @code{1*10^2+3*10^1+2*10^0}, which results in a different value:
132.  When we consider other numerical bases, the bases in the
polynomial change accordingly, but the correspondence between written
form and value stands: for example, the value of 0x123 is calculated
as @code{1*16^2+2*16^1+3*16^0}.

The ``higher'' a digit is in the polynomial, the @dfn{more
significant} it is, @i{i.e.} the more weight it has on the value of the
number where it appears.  In the written number @code{123}, for
example, the digit 1 is the @dfn{most significant} digit of the
number, and the digit 3 is the @dfn{least significant} digit.

This distinction between the written form of a number and its value is
very important.  Just like in certain languages letters are read
right-to-left (Arabic) or even down-to-up (Japanese) we could
certainly conceive a language in which the digits of numbers were
arranged from right-to-left instead of left-to-right.  In such a
language the written representation of 123 would be @code{321}, not
@code{123}.  In other words: the least significant digit would come
first, not last, in the written form of the number.

Now when it comes to store numbers in computers, rather than writing
them on a paper, the role of the paper is played by the computer's
memory, be it ephemeral (like RAM) or persistent (like a spinning hard
disk or a Flash memory), which is organized as a sequence of bytes.
Since we are composing numbers with bytes, it makes sense to have each
byte to play the role of a digit in the written form of the bigger
number.  Since bytes can have values from 0 to 255, the base is 256.
But what is the ``written form'' for our byte-composed numbers?

In the last section we tried to compose bigger integers by
concatenating bytes together and interpreting the result.  In doing
so, we assumed (quite naturally) that in the written form of the
resulting integer the bytes are ordered in the same order than they
appear in the file, @i{i.e.} we assume that the written form of the number
@code{b1*256^2+b2*256^1+b3*256^0} would be @code{b1b2b3}, where
@code{b1}, @code{b2} and @code{b3} are bytes.  In other words, given a
written form @code{b1b2b3}, @code{b1} would be the most significant
byte (digit) and @code{b3} would be the least significant byte
(digit).  In our world of IO spaces, the ``written form'' is the
disposition of the bytes in the IO space (file, memory buffer, @i{etc})
being edited.

That interpretation of the written form is exactly what the
bit-concatenation operator implements:

@example
(poke) dump :from 0#B :size 3#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c                                  .EL
(poke) var b1 = byte @@ 0#B
(poke) var b2 = byte @@ 1#B
(poke) var b3 = byte @@ 2#B
(poke) b1:::b2:::b3
(uint<24>) 0x7f454c
@end example

However, much like in certain human languages the written form is read
from right to left, some computers also read numbers from right to
left in their ``written form''.  Actually, turns out that @emph{most}
modern computers do it like that.  This means that, in these
computers, given the written form @code{b1b2b3} (@i{i.e.} given a file
where @code{b1} comes first, followed by @code{b2} and then @code{b3})
the most significant byte is @code{b3} and the least significant byte
is @code{b1}.  Therefore, the value of the number would be
@code{b3*256^2+b2*256^1+b3*256^0}.

So, given the written form of a bigger number @code{b1b2b3} (@i{i.e.} some
ordering of bytes implied by the file they are stored in) there are at
least two ways to interpret them to calculate the value of the number.
When the written form is read from left to right, we talk about a
@dfn{big endian} interpretation.  When the written form is read from
right to left, we talk about a @dfn{little endian} interpretation.

Given the first three bytes in @code{foo.o}, we can determine the
value of the integer composed of these three bytes in both
interpretations:

@example
(poke) b1:::b2:::b3
(uint<24>) 0x7f454c
(poke) b3:::b2:::b1
(uint<24>) 0x4c457f
@end example

Remember how the type specifier @code{byte} is just a synonym of
@code{uint<8>}, and how we can use type specifiers like
@code{uint<24>} and @code{uint<32>} to map bigger integers?  When we
do that, like in:

@example
(poke) uint<24> @@ 0#B
(uint<24>) 0x7f454c
@end example

@noindent
Poke should somehow decide what kind of interpretation to use,
@i{i.e.} how to read the ``written form'' of the number.  As you can see
from the example, poke uses the left-to-right interpretation, or
big-endian, by default.  But you can change it using a new
dot-command: @command{.set endian}:

@example
(poke) .set endian little
(poke) uint<24> @@ 0#B
(uint<24>) 0x4c457f
@end example

The currently used interpretation (also called @dfn{endianness}) is
shown if you invoke the dot-command without an argument@footnote{This
also applies to the other @command{.set} commands}:

@example
(poke) .set endian
little
@end example

Different systems use different endianness.  Into a given system, it
is to be expected that most files will be encoded following the same
conventions.  Therefore poke provides you a way to set the endianness
to whatever endianness is in the system.  You do it this way:

@example
(poke) .set endian host
@end example

@node Negative Integers
@section Negative Integers

@subsection Negative Encodings

Up to this point we have worked with unsigned integers, @i{i.e.} whole
numbers which are zero or bigger than zero.  Much like it happened
with endianness, the interpretation of the value of several bytes as a
negative number depends on the specific interpretation.

@cindex negative encoding
@cindex one complement
@cindex two complement
In computing there are two main ways to interpret the values of a
group of bytes as a negative number: @dfn{one's complement} and
@dfn{two's complement}.

At the moment GNU poke supports the two complement interpretation,
which is really ubiquitous and is the negative encoding used by the
vast majority of modern computers and operating systems.

We may consider adding support for one's complement in the future, but
only if there are real needs that would justify the effort (which
wouldn't be a small one ;)).

@subsection Signed Integers

Unsigned values are never negative.  For example:

@example
(poke) 0UB - 1UB
0xffUB
@end example

@noindent
Instead of getting a -1, we get the result of an unsigned underflow,
which is the biggest possible value for an unsigned integer of size 8
bits: 0xff.

When using type specifiers like @code{uint<8>} or @code{uint<16>} in a
map, we get unsigned values such as 0UB.  We need other type
specifiers to map signed values.  These look like @code{int<8>} and
@code{int<16>}.

For example, let's map a signed 16-bit value from @file{foo.o}:

@example
(poke) .set obase 10
(poke) int<16> @@ 0#B
28515H
@end example

@noindent
Note how the suffix of the value is now @code{H} and not @code{UH}.
This means that the value is signed!  But in this case the mapped
integer is still positive, so let's try to get an actual negative
value:

@example
(poke) var h = int<16> @@ 0#B
(poke) h - h - 1H
-1H
@end example

@subsection Mixing Signed and Unsigned Integers

Adding two signed integers gives you a signed integer:

@example
(poke) 1 + 2
3
@end example

@noindent
Likewise, adding two unsigned integers results in an unsigned integer:

@example
(poke) 1U + 2U
3U
@end example

@noindent
But, what happens if we mix signed and unsigned values in an
expression?  Is the result signed, or unsigned?  Let's find out:

@example
(poke) 1U + 2
3U
@end example

@noindent
Looks like combining an unsigned value with a signed value gives us an
unsigned value.  This actually applies to all the operators that work
on integer values: multiplication, division, exponentiation, @i{etc}.

@cindex casts
What actually happens is that the signed operand is converted to an
unsigned value before executing the expression.  You can also convert
signed values into unsigned values (and vice-versa) using @dfn{cast
constructions}:

@example
(poke) 2 as uint<32>
2U
@end example

@noindent
Therefore, the expression @code{1U + 2} is equivalent to @code{1U + 2
as uint<32>}:

@example
(poke) 1U + 2 as uint<32>
3U
@end example

You may be wondering: why not doing it the other way around?  Why not
converting the unsigned operand into a signed value and then operate?
The reason is that, given an integer of some particular size, the
positive range that you can store in it is bigger when interpreted as
an unsigned integer than when interpreted as a signed integer.
Therefore, converting signed into unsigned before operating reduces
the risk of positive overflow, which by the way is not allowed in
poke.  This of course assumes that we, as users, will be working with
positive numbers more often than with negative numbers, but that is a
reasonable assumption to do, as it is often the case!

@node Weird Integers
@section Weird Integers

Up to this point we have been playing with integers that are built
using a whole number of bytes.  However, we have seen that the type
specifier for an integer has the form @code{int<N>} or @code{uint<N>}
for signed and unsigned variants, where @code{N} is the width of the
integer, in bits.  We have used bit-sizes that are multiple of 8,
which is the size of a byte.  So, why is this so?  Why is @code{N} not
measured in bytes instead?

The reason is that poke is not limited to integers composed of a whole
number of bytes. You can actually have integers composed of @emph{any}
number of bits, between 1 and 64.  So yes, @code{int<3>} is a type
specifier for signed 3-bit integers, and @code{uint<17>} is a type
specifier for unsigned 17-bit integers.

We call integers like this @dfn{weird integers}.

The vast majority of programming languages do not provide any support
for weird integers.  In the few cases they do, it is often in a very
limited and specific way, like bitmap fields in C structs.  Such
constructions are often vague, obscure, and often their semantics
depend on the specific implementation of the language, and/or the
characteristics of the system where you run your program.

In poke, on the contrary, weird numbers are first class citizens, and
they don't differ in any way from ``normal'' integers which are
composed of a whole number of bytes.  Their interpretation is also
well defined, and they keep the same semantics regardless of the
characteristics of the computer on which poke is running.

@subsection Incomplete Bytes

Let's consider first weird numbers that span for more than one byte.
For example, an unsigned integer of 12 bits.  Let's visualize the
written form of this number, @i{i.e.} the sequence of its constituent
bytes as they appear in the underlying IO space:

@example
  byte 0  |  byte 1
+---------+----+----+
|::::::::::::::|    |
+---------+----+----+
|   uint<12>   |
@end example

@noindent
All right, the first byte is used in its entirely, but only half of
the second byte is used to conform the value of the number.  The other
half of the second byte has no influence of the value of the 12 bits
number.

Now, we talk about the ``second half of the byte'', but what do that
means exactly?  We know that bytes in memory and files (bytes in IO
spaces) are indivisible at the system level: bytes are read and
written one at a time, as little integers in the range @code{0..255}.
However, we can create the useful fiction that each byte is composed
by @dfn{bits}, which are the digits in the binary representation of
the byte value.

So, we can look at a byte as composed of a group of eight bits, like
this:

@example
           byte
+-------------------------+
| b7 b6 b5 b4 b3 b2 b1 b0 |
+-------------------------+
@end example

@noindent
Note how we decided to number the bits in descending order from left
to right.  This is because these bits correspond to the base of the
polynomial equivalent to the binary value of the byte, @i{i.e.} the
value of the byte is
@code{b7*2^7+b6*2^6+b5*2^5+b4*2^4+b3*2^3+b2*2^2+b1*2^1+b0*2^0}.  In
other words: at the bit level poke always uses a big endian
interpretation, and the bit that ``comes first'' in this imaginary
stream of bits is the most significant bit in the binary
representation of the number.  Please note that this is just a
convention imposed by the poke authors: the opposite could have been
chosen, but it would have been a bit confusing, as we would have to
picture binary numbers in reverse order!

With this new way of looking at bytes, we can now visualize what we
mean exactly with the ``first half'' and ``second half'' of the
trailing byte, in our 12 bits unsigned number:

@example
           byte 0         |           byte 1
+-------------------------+-------------+-------------+
| a7 a6 a5 a4 a3 a2 a1 a0   b7 b6 b5 b4 :             |
+-------------------------+-------------+-------------+
|                uint<12>               |
@end example

@noindent
Thus the first half of @code{byte 1} is the sequence of bits @code{b7
b6 b5 b4}.  The second half, which is not pictured since it doesn't
contribute to the value of the number, would be @code{b3 b2 b1 b0}.

So what would be the value of the 12-bit integer?  Exactly like with
non-weird numbers, this depends on the current selected endianness,
which determines the ordering of bytes.

If the current endianness is big, then @code{byte 0} provides the most
significant bits of the result number, and the used portion of
@code{byte 1} provides the least significant bits of the result
number:

@example
0b a7 a6 a5 a4 a3 a2 a1 a0 b7 b6 b5 b4
@end example

@noindent
However, if the current selected endianness is little, then the used
portion of @code{byte 1} provides the most significant bits of the
result number, and @code{byte 0} provides the least significant bits
of the result number:

@example
0b b7 b6 b5 b4 a7 a6 a5 a4 a3 a2 a1 a0
@end example

Let's see this in action.  Let's take a look to the value of the first
two bytes in @file{foo.o}, in binary:

@example
(poke) .set obase 2
(poke) byte @@ 0#B
0b01111111UB
(poke) byte @@ 1#B
0b01000101UB
@end example

@noindent
Looking at these bytes as sequences of bits, we have:

@example
        byte @@ 0#B       |        byte @@ 1#B
+-------------------------+-------------+-------------+
|  0  1  1  1  1  1  1  1    0  1  0  0 :  0  1  0  1
+-------------------------+-------------+-------------+
|                uint<12>               |
@end example

@noindent
Let's map our weird number at offset 0 bytes, using big endian:

@example
(poke) .set endian big
(poke) uint<12> @@ 0#B
(uint<12>) 0b011111110100
@end example

@noindent
That matches what we explained before: the most significant bits of
the unsigned 12 bits number come from the byte at offset 0,
@i{i.e.} @code{01111111}, whereas the least significant bits come from the
byte at offset 1, @i{i.e.} @code{0100}.

Now let's map it using little endian:

@example
(poke) uint<12> @@ 0#B
(uint<12>) 0b010001111111
@end example

@noindent
This time the most significant bits of the unsigned 12 bits number
come from the byte at offset 1, @i{i.e.} @code{0100}, whereas the least
significant bits come from the byte at offset 0, @i{i.e.} @code{01111111}.

An important thing to note is that non-weird numbers, @i{i.e.} numbers
built with a whole number of bytes, are basically a particular case of
weird numbers where the last byte in the written form (in the IO
space) provides all its bits.  The rules are exactly the same in all
cases, which makes it easy to obtain predictable and natural results
when building integers using poke.

@subsection Quantum Bytenics

The second kind of weird numbers are integers using less than 8 bits.
These ``sub-byte'' numbers do not use all the bits of their containing
byte.  Consider for example the written form of an unsigned integer of
size 5 bits:

@example
    byte
+-----+----+
|:::::|    |
+-----+----+
uint<5>
@end example

@noindent
Now let's view the byte as a sequence of bits:

@example
             byte
+----------------+----------+
| b7 b6 b5 b4 b3 |          |
+----------------+----------+
|    uint<5>     |
@end example

@noindent
What is the value of this number?  Applying the general rules for
building integers from bytes, we can easily see that regardless of the
current endianness the value, in binary, is:

@example
0b b7 b6 b5 b4 b3
@end example

Let's see this in poke:

@example
(poke) .set obase 2
(poke) .set endian big
(poke) byte @@ 0#B
0b01111111UB
(poke) uint<5> @@ 0#B
(uint<5>) 0b01111
(poke) .set endian little
(poke) uint<5> @@ 0#B
(uint<5>) 0b01111
@end example

@subsection Signed Weird Numbers

In the section discussing negative integers, we saw how the difference
between a signed number and an unsigned number is basically a
different interpretation of the most significant bit.  Exactly the
same applies to weird numbers.

Let's summon our unsigned 12-bit integer at the beginning of the file
@file{foo.o}:

@example
(poke) .set endian big
(poke) uint<12> @@ 0#B
(uint<12>) 0b011111110100
@end example

@noindent
The most significant bit of the resulting value (not of its written
form) indicates that this number would be positive if we were mapping
the corresponding signed value.  Let's see:

@example
(poke) int<12> @@ 0#B
(int<12>) 0b010001111111
(poke) .set obase 10
(poke) int<12> @@ 0#B
(int<12>) 1151
@end example

Let's make it a bit more interesting, and change the value of the first
byte in the file so we get a negative number:

@example
(poke) .set obase 2
(poke) byte @@ 0#B = 0b1111_1111
(poke) int<12> @@ 0#B
(int<12>) 0b111111110100
(poke) .set obase 10
(poke) int<12> @@ 0#B
(int<12>) -12
@end example

@noindent
Now, let's switch to little endian:

@example
(poke) .set endian little
(poke) .set obase 2
(poke) int<12> @@ 0#B
(int<12>) 0b010011111111
(poke) .set obase 10
(poke) int<12> @ 0#B
(int<12>) 1279
@end example

@node Unaligned Integers
@section Unaligned Integers

@cindex IO devices
@cindex IO spaces
We have mentioned above that the data stored in computers, that we
edit with poke, is arranged as a sequence of bytes.  The entities we
edit with poke (that we call @dfn{IO devices}) are presented to us as
IO spaces.  Up to now, we have accessed this IO space in terms of
bytes, in commands like @code{dump :from 32#B} and in expressions like
@code{2UB + byte @@ 0#B}.  We said that mapped integers are built from
bytes read from the IO space.

However, the IO space that poke offers to us is actually a space of
bits, not a space of bytes, and the poke values are mapped on this
space of bits.  The following figure shows this:

@example
poke values      |        uint<16> @@ 2#b        |
-----------      |                               |
IO space     |b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|
-----------  |               |               |               |
IO device    |     byte0     |     byte1     |     byte2     |
@end example

@noindent
The main consequence of this, that you can see in the figure above, is
that we can use offsets in mapping operations that are not
@dfn{aligned to bytes}.  You can specify an offset in bits, instead of
bytes, using the @code{#b} suffix instead of @code{#B}.  Little
@code{b} means bits, and big @code{B} means bytes.

Let's map an unaligned 16 bit unsigned integer in @file{foo.o}:

@example
(poke) dump :from 0#B :size 3#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c                                  .EL
(poke) .set obase 2
(poke) byte @@ 0#B
0b01111111UB
(poke) byte @@ 1#B
0b01000101UB
(poke) byte @@ 2#B
0b01001100UB
(poke) .set endian big
(poke) uint<16> @@ 2#b
0b1111110100010101UH
@end example

Graphically:

@example
poke values      |        uint<16> @@ 2#b        |
-----------      |                               |
IO space     |0|1|1|1|1|1|1|1|0|1|0|0|0|1|0|1|0|1|0|0|1|1|0|0|
-----------  |               |               |               |
IO device    |     0x7f      |      0x45     |      0x4c     |
@end example

These three levels of abstractions make it very easy and natural to
work with unaligned data.  Imagine for example that you are poking
packages in a network protocol that is bit-oriented.  This means that
the packages will generally not be aligned to byte boundaries, but
still the payload stored in the packages contains integers of several
sizes.  Other conventional binary editors or programming languages,
that are almost always byte oriented, would require us to ``unpack''
the network data to a different, byte oriented, representation before
messing with it.  poke, on the contrary, allows you to directly map
these integers as if they were aligned to byte boundaries, and work
with them.

However, when one tries to determine the correspondence between a
given poke value and the underlying bytes in the IO device, things can
get complicated.  This is particularly true when we map what we called
``weird numbers'', @i{i.e.} numbers with partial bytes.  As we saw, the
rules to build these numbers were expressed in terms of bytes.

In order to ease the visualization of the process used to build
integer values (especially if they are weird numbers, @i{i.e.} integers
with partial bytes) one can imagine an additional layer of ``virtual
bytes'' above the space of bits provided by the IO space.
Graphically:

@example
poke values       |        uint<16> @@ 2#b        |
-----------       |                               |
Virtual bytes     | virt. byte1   |  virt. byte2  |
-----------       |               |               |
IO space      |0|1|1|1|1|1|1|1|0|1|0|0|0|1|0|1|0|1|0|0|1|1|0|0|
-----------   |               |               |               |
IO device     |     0x7f      |      0x45     |      0x4c     |
@end example

It is very important to understand that the IO space is an abstraction
provided by poke.  The underlying file, or memory buffer, or whatever,
is actually a sequence of bytes; poke translates the operations on
integers, bits, bytes, @i{etc} into the corresponding byte operations,
and this translation is far from trivial.  Fortunately, we can let
poke do the dirty job for us, and abstract ourselves from that
complexity.

@node Integers of Different Sizes
@section Integers of Different Sizes

When integer values of different sizes are passed to an arithmetic or
relational operator, the ``smaller'' operand gets converted into the
size of the ``bigger'' operand.  For example:

@example
(poke) 1H + 2
3
@end example

@noindent
The operands are of size 16-bit and 32-bit respectively, and the
result is a 32-bit integer.  This is equivalent to:

@example
(poke) 1H as int<32> + 2
3
@end example

@node Offsets and Sizes
@section Offsets and Sizes

Early in the design of what is becoming GNU poke I was struck by a
problem that, to my surprise, would prove not easy to fix in a
satisfactory way: would I make a byte-oriented program, or a
bit-oriented program?  Considering that the program in question was
nothing less than an editor for binary data, this was no petty
dilemma.

Since the very beginning I had a pretty clear idea of what I wanted to
achieve: a binary editor that would be capable of editing user defined
data structures, besides mere bytes and bits.  I also knew I needed
some sort of domain specific language to describe these structures and
operate on them.  How that language would look like, and what kind of
abstractions it would provide, however, was not clear to me.  Not at
all.

So once I sketched an initial language design, barely something very
similar to C structs, I was determined to not continue with the poke
implementation until I had described as many as binary formats in my
language as possible.  That, I reckoned, was the only way to make sure
the implemented language would be expressive, complete and useful
enough to fulfil my requirements.

The first formats I implemented using my immature little language
included ELF, FLV, MP3, BSON@dots{} all of them describing structures
based on whole bytes.  Even when they try to be compact, it is always
by packing bit-fields in elements that are, invariably, sized as a
multiple of bytes.  Consequently, the language I was evolving became
byte oriented as well.  No doubt also influenced by my C inheritance,
I would think of bit-fields either as a sort of second class citizen,
or as mere results of shifting and masking.

This worked well.  The language evolved to be able to express many
different aspects of these formats in a very nice way, like
variable-length data and holes in structures.

Consider the following definition, which is @strong{not} valid in
today's Poke:

@example
type Data =
  struct
  @{
    byte magic;
    byte count;
    byte dstart;

    byte[count] data @@ dstart;
  @};
@end example

The data starts with a byte that is a magic number.  Then the size of
the data stored, in bytes, and then the data itself.  This data,
however, doesn't start right after @code{dstart}: it starts at
@code{dstart}, which is expressed as an offset, in bytes, since the
beginning of the Data.  I conceived struct field labels to be any
expression evaluating to an integer, which would be interpreted as a
number of bytes, obviously.

@cindex deflate
Then, one day, it was the turn for IETF RFC1951, which is the
specification of the DEFLATE algorithm and associated file format.  Oh
dear.  Near the beginning of the spec document it can be read:

@quotation
This document does not address the issue of the order in which bits of
a byte are transmitted on a bit-sequential medium, since the final
data format described here is byte- rather than bit-oriented.
However, we describe the compressed block format in below, as a
sequence of data elements of various bit lengths, not a sequence of
bytes.
@end quotation

Then it goes on describing rules to pack the DEFLATE elements into
bytes.  I was appalled, and certainly sort of deflated as well.  The
purpose of my program was precisely to edit binary in terms of the
data elements described by a format.  And in this case, these data
elements came in all sort of bit lengths and alignments.  This can be
seen in the following RFC1951 excerpt, that describes the header of a
compressed block:

@quotation
Each block of compressed data begins with 3 header bits
containing the following data:
@example
first bit       BFINAL
next 2 bits     BTYPE
@end example

Note that the header bits do not necessarily begin on a byte boundary,
since a block does not necessarily occupy an integral number of bytes.
@end quotation

At this point I understood that my little language on the works would
never be capable to describe the DEFLATE structures naturally: C-like
bit-fields, masking and shifting, all based on byte-oriented
containers and boundaries, would never provide the slickness I wanted
for my editor.  I mean, just use C and get done with it.

This pissed me off.  Undoubtedly other formats and protocols would be
impacted in a similar way.  Even when most formats are byte oriented,
what am I supposed to tell to the people hacking bit-oriented stuff?
``Sorry pal, this is not supported, this program is not for you''?  No
way, I thought, not on my watch.

The obvious solution for the problem, is to be general.  In this case,
to express every offset and every memory size in bits instead of
bytes.  While this obviously gives the language maximum expressiveness
power, and is ideal for expressing the few bit-oriented formats, it
has the disadvantage of being very inconvenient for most situations.

To see how annoying this is, let's revisit the little @code{BitData}
structure we saw above.  In a bit-oriented description language, we
would need to write something like:

@example
type BitData =
  struct
  @{
    byte magic;
    byte count;
    byte dstart;

    byte[count] data @@ dstart * 8;
  @};
@end example

Yeah@dots{} exactly.  The @key{*} and @key{8} keys in the keyboards of
the poke users would wear out very fast, not to mention their patience
as well.  Also, should I provide both @code{sizeof} and
@code{bitsizeof} operators?  Nasty.

I am very fond of the maxim ``Never write a program you would never
use yourself''@footnote{Actually it is Lord Vetinari's ``Never build a
dungeon you can't get out of.'' but the point is the same.}, so I
resigned myself to make GNU poke byte oriented, and to provide as many
facilities for operating on bit-fields as possible.

@noindent
Fortunately, I have smart friends@dots{}

@cindex rabbit herd
During one of the Rabbit Herd's Hacking
Weekends@footnote{@url{http://www.jemarch.net/rhhw}} I shared my
frustration and struggle with the other rabbits, and we came to
realize that offsets and data sizes in Poke should not be pure
magnitudes or mere integer values: they should be united.  They should
have units.

It makes full sense when you come to think about it.  For a program
like poke, it is only natural to talk about different memory units,
like bits, bytes, kilobytes, megabits, and so on.  Bits and bytes are
just two common units.

The syntax that we eventually used to denote united values is to
specify the @dfn{magnitude} part, a dash (@code{'#'} character, and
then the @dfn{unit}.  For example, the size ``three bytes'' is
expressed as @code{3#B}, and ``three bits'' as @code{3#b}.

Apart from allowing me to express values in different units, this
approach also has other benefits as we will see shortly.

I'm really grateful to Bruno Haible, Luca Saiu and Nacho Gonzalez for
putting me on the right track.

@node Buffers as IO Spaces
@section Buffers as IO Spaces

We have mentioned already that files are not the only entities that
can be edited using poke. Remember the dot-command @command{.file}
that opened a file as an IO space?

@example
(poke) .file foo.o
The current IOS is now `./foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #0	FILE	rw	0x00000000#B	0x000004c8#B	./foo.o
@end example

@noindent
Memory buffers can be created using a similar dot-command,
@command{.mem}:

@example
(poke) .mem foo
The current IOS is now `*foo*'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	MEMORY		0x00000000#B	0x00001000#B    *foo*
  #0	FILE	rw	0x00000000#B	0x000004c8#B    ./foo.o
@end example

Note how the name of the buffer is built by prepending and appending
asterisks.  Therefore, the name of the buffer created by the command
@code{.mem foo} is @code{*foo*}.  Note also that the new buffer is
created with a default size of 0x1000 bytes, or 4096 bytes.  The
contents of the buffer are zeroed:

@example
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

Memory buffer IO spaces grow automatically when a value is mapped
beyond their current size.  This is very useful when populating newly
created buffers.  However, for security reasons, there is a limit: the
IO spaces are only allow to grow 4096 bytes at a time.

When it comes to map values, there is absolutely no difference between
an IO space backed by a file and an IO space backed by a memory
buffer.  Exactly the same rules apply in both cases.

@node Copying Bytes
@section Copying Bytes

Memory buffer IO spaces are cheap, and they are often used as
``scratch'' areas.

Suppose for example we want to experiment with the ELF header of
@file{foo.o}.  We could open it in poke:

@example
(poke) .file foo.o
The current IOS is now `./foo.o'.
@end example

@noindent
The header of an ELF file comprises the first 64 bytes in the
file:

@example
(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 3e00 0100 0000 0000 0000 0000 0000  ..>.............
00000020: 0000 0000 0000 0000 0802 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0b00 0a00  ....@.....@.....
@end example

We know that as soon as we poke something on an IO space, the
underlying file is immediately modified.  So if we start playing with
@file{foo.o}'s header, we may corrupt the file.  We could of course
make a copy of @file{foo.o} and work on the copy, but it is much
better to create a memory IO space and copy the ELF header there:

@example
(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) copy :from_ios 0 :from 0#B :size 64#B
(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 3e00 0100 0000 0000 0000 0000 0000  ..>.............
00000020: 0000 0000 0000 0000 0802 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0b00 0a00  ....@.....@.....
@end example

@noindent
The command @command{copy} above copies 64 bytes starting at byte 0
from the IO with id 0 (the file @file{foo.o}) to the current IO space
(the buffer @code{*scratch*}).

Once we are done working with the copy of the ELF header, and
satisfied, we can copy it back to the file and close the memory IO
space:

@example
(poke) copy :from 0#B :size 64#B :to_ios 0
(poke) .close
The current IOS is now `./foo.o'.
@end example

@noindent
Note how the command arguments @code{:from_ios} and @code{:to_ios} are
assumed to be the current IO space if they are not explicitly
specified in the command invocation.  For detailed information on the
@command{copy} command, see @ref{copy}.

@node Saving Buffers in Files
@section Saving Buffers in Files

Another useful command when working with buffer IO spaces (and in
general, with any IO space) is @command{save}.  Let's say we want to
save a copy of the header of an ELF file in another file.  We could do
it the pokeish way:

@example
$ poke foo.o
[@dots{}]
(poke) save :from 0#B :size 64#B :file "header.dat"
@end example

@noindent
The command above saves the first 64 bytes in the current IO space
(which is backed by the file @file{foo.o}) into a new file
@file{header.dat} in the current working directory.

Let's now consider again the scenario where we are using a memory IO
space as a scratch area.  It is late in the night and we are tired, so
we would like to save the contents of the scratch buffer to a file, so
we can continue our work tomorrow.  This is how we would do that:

@example
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	MEMORY		0x00000000#B	0x00001000#B    *scratch*
  #0	FILE	rw	0x00000000#B	0x000f4241#B    ./foo.o
(poke) save :from 0#B :size iosize (1) :file "scratch.dat"
@end example

@noindent
Here we used the built-in function @code{iosize}, that given an IO
space identifier returns its size.

@node Character Sets
@section Character Sets

@cindex character
@cindex character set
Computers understand text as a sequence of @dfn{codes}, which are
numbers identifying some particular @dfn{character}.  A character can
represent things like letters, digits, ideograms, word separators,
religious symbols, @i{etc}.  Collections of character codes are called
@dfn{character sets}.

@cindex ASCII
@cindex Latin-1
Some character sets try to cover one or a few similar written
languages.  This is the case of ASCII and ISO Latin-1, for example.
These character sets are small, @i{i.e.} just a few hundred codes.

@cindex Unicode
Other character sets are much more ambitious.  This is the case of
Unicode, that tries to cover the entire totality of human languages in
the globe, including the fictitious ones, like klingon.  Unicode is a
really big character set.

In order to store character codes in a computer's memory, or a file,
we need to @dfn{encode} each character code in one or more bytes.  The
number of bytes needed to encode a given character code depends on the
range of codes in the containing set.

ASCII, for example, defines 128 character codes: a single byte is
enough to encode every possible ASCII character.  It is very easy to
encode ASCII.

@cindex utf-8
Unicode, on the contrary, defines many thousand of character codes,
and has room for many more: we would need 31 bits in order to encode
any conceivable Unicode character code.  However, it would be wasteful
to use that many bits per character: most used character codes tend to
be in lower regions of the code space.  For example, the code
corresponding to the Latin letter @code{'a'} is a fairly small number,
whereas the codes corresponding to the Klingon alphabet are really big
numbers.  Consequently, some systems opt to just encode a subset of
Unicode, like the first 16 bits of the Unicode space, which is called
the Basic Multilingual Plane, and contains all the characters that
most people will ever need.  There are also variable-length encodings
of Unicode, that try to use as less bytes as possible to encode any
given code.  A particularly clever encoding of Unicode, designed by
Rob Pike, is backwards compatible with the ASCII encoding, @i{i.e.} it
encodes all the ASCII codes in one byte each, and the values of these
byte are the same than in ASCII.  This clever encoding is called
UTF-8.

@node From Bytes to Characters
@section From Bytes to Characters

@subsection Character Literals

poke has built-in support for ASCII, and its simple encoding: each
ASCII code is encoded using one byte.  Let's see:

@example
(poke) 'a'
0x61UB
@end example

@noindent
We presented poke with the character @code{a}, and it answered with
its corresponding code in the ASCII character set, which is 0x61.  In
fact, 'a' and 0x61UB are just two ways to write exactly the same byte
value in poke:

@example
(poke) 'a' == 0x61UB
1
(poke) 'a' + 1
0x62U
@end example

In order to make this more explicit, poke provides yet another synonym
for the type specifier @code{uint<8>}: @code{char}.

@subsection Classifying Characters

When working with characters it is very useful to have some
acquaintance of the ASCII character set, which is designed in a very
clever way with the goal of easing certain code calculations.  See
@ref{Table of ASCII Codes} for a table of ASCII codes in different
numeration bases.

Consider for example the problem of determining whether a byte we map
from an IO space is a digit.  Looking at the ASCI table, we observe
that digits all have consecutive codes, so we can do:

@example
(poke) var b = byte @@ 23#B
(poke) b >= '0' && b <= '9'
1
@end example

Now that we know that @code{b} is a digit, how could we calculate its
digit value?  If we look at the ASCII table again, we will find that
the character codes for digits are not only consecutive: they are also
ordered in ascending order @code{0}, @code{1}, @dots{}  Therefore, we can
do:

@example
(poke) b
0x37UB
(poke) b - '0'
7UB
@end example

@noindent
@code{b} contains the ASCII code 0x37UB, which corresponds to the
character @code{7}, which is a digit.

How would we check whether @code{b} is a letter?  Looking at the ASCII
table, we find that lower-case letters are encoded consecutively, and
the same applies to upper-case letters.  This leads to repeat the
trick again:

@example
(poke) (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z')
0
@end example

@subsection Non-printable Characters

Not all ASCII code are printable using the glyph that are usually
supported in terminals.  If you look at the table in @ref{Table of
ASCII Codes}, you will find codes for characters described as ``start
of text'', ``vertical tab'', and so on.

These character codes, which are commonly known as @dfn{non-printable
characters}, can be represented in poke using its octal code:

@example
(poke) '\002'
0x2UB
@end example

@noindent
This is of course no different than using @code{2UB} directly, but in
some contexts the ``character like'' notation may be desirable, to
stress the fact that the byte value is used as an ASCII character.

@cindex escape sequence
Some of the non-printable characters also have alternative notations.
This includes new-line and horizontal tab:

@example
(poke) '\n'
0xaUB
(poke) '\t'
0x9UB
@end example

@noindent
These @code{\} constructions in character literals are called
@dfn{escape sequences}. See @ref{Characters} for a complete list of
allowed escapes in character literals.

@node ASCII Strings
@section ASCII Strings

@subsection String Values

Now that we know how to manipulate ASCII codes in poke, we may wonder
how can we combine them to conform words or, more generally,
@dfn{strings} of ASCII characters.

GNU poke has support for native ASCII string values.  The characters
conforming the string are written between @code{"} and @code{"}
characters, like in:

@example
(poke) "word"
"word"
@end example

@noindent
Note, and this is important, that string values are as atomic as
integer values: they are not really composite values.  The fact that
@code{"word"} contains an @code{r} at position 3 is like the fact that
the value @code{123} contains a digit @code{2} at position 2.

Like in character literals, poke strings support several escape
sequences that help to denote non-printed characters, such as new
lines and horizontal tabs.  @xref{String Literals}.

@subsection Poking Strings

Let's start with a fresh memory buffer IOS @code{*scratch*}:

@example
(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

@noindent
If we wanted to, somehow, store the word @code{word} in this IO space,
encoded in ASCII, we could proceed as:

@example
(poke) char @@ 0x12#B = 'w'
(poke) char @@ 0x13#B = 'o'
(poke) char @@ 0x14#B = 'r'
(poke) char @@ 0x15#B = 'd'
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 776f 7264 0000 0000 0000 0000 0000  ..word..........
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

@noindent
This worked.  The ASCII part of the @command{dump} output, which
interprets the bytes as ASCII, clearly shows the word @code{word} at
the offset where we poked the character values.  However, we can do
better: string values can be mapped themselves.

String values use the type specifier @code{string}.  As any other kind
of value in poke, they can be mapped from an IO space:

@example
(poke) string @@ 0x12#B
"word"
@end example

@noindent
Clearly that is the string resulting from the concatenation of the
character values that we poked before.

The question now is: how did poke know that the last character of the
string was the @code{d} at offset 0x15#B?  The fact the character code
0 (also known as the @dfn{NULL character}) at offset 0x16#B is
non-printable, doesn't imply it is not part of the ASCII character
set.  Clearly, we have to pick an ASCII code and reserve it to mark
the end of strings.  Like the C programming language, and countless
formats and systems do, poke uses the NULL character to delimit
strings.

Now consider this:

@example
(poke) "word"'length
4UL
(poke) "word"'size
40UL#b
@end example

@noindent
Using the @code{length} and @code{size} attributes, poke tells us that
the length of the string @code{"word"} is 4 characters, but the size
of the string value is 40 bits, or 5 bytes.  Why this discrepancy?
The @code{size} value attribute tells how much storage space a given
value required once mapped to an IO space, and in the case of strings
it should count the space occupied by the terminating NULL character.

Poking string values on the IO space is as straightforward as poking
integers:

@example
(poke) string @@ 0x22#B = "WORD"
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 776f 7264 0000 0000 0000 0000 0000  ..word..........
00000020: 0000 574f 5244 0000 0000 0000 0000 0000  ..WORD..........
@end example

@subsection From Characters to Strings

Strings can be concatenated using the string-concatenation @code{+}
operators:

@example
(poke) "foo" + "bar"
"foobar"
@end example

@noindent
The string resulting from the operation above has length 6 characters,
and size 7 bytes.  The terminating NULL character of @code{"foo"} is
lost in the operation.  This is easily seen:

@example
(poke) "foo"'size + "bar"'size
0x40UL#b
(poke) ("foo" + "bar")'size
0x38UL#b
@end example

The string concatenation operator requires two strings.  Therefore, if
we wanted to append a character to a string, we would get an error:

@example
(poke) "Putin" + 'a'
<stdin>:1:1: error: invalid operands in expression
"Putin" + 'a';
^~~~~~~~~~~~~
@end example

@noindent
It is possible to transform a character value (@i{i.e.} a byte value) into
a string composed of that character using a cast:

@example
(poke) 'a' as string
"a"
@end example

@noindent
Using that cast, we can now append:

@example
(poke) "Putin" + 'a' as string
"Putina"
@end example

@node From Strings to Characters
@section From Strings to Characters

Despite being atomic values, poke strings can be indexed in order to
retrieve individual characters:

@example
(poke) "word"[2]
0x72UB
@end example

@noindent
Note how the indexing is zero-based, @i{i.e.} the first position in the
string is referred as @code{[0]}, the second position with @code{[1]},
and so on.

If you specify a negative index, or an index that is too big, you will
get an error:

@example
(poke) "word"[-1]
<stdin>:1:8: error: index is out of bounds of string
"word"[-1];
       ^
(poke) "word"[5]
<stdin>:1:8: error: index is out of bounds of string
"word"[5];
       ^
@end example

@node Strings are not Arrays
@section Strings are not Arrays

Arrays are collections of homogeneous data organized in a sequence.
We have already seen (briefly) arrays of bytes, which use the type
specifier @code{byte[]}.

Similarly, it is possible to arrange characters (which are basically
little numbers) in an array, like in:

@example
(poke) ['a','b','c']
[0x61UB,0x62UB,0x63UB]
@end example

@noindent
However, the array above is @emph{not} equivalent to the string
@code{"abc"}.  The later is a simple value, whereas an array is a
composite value, and also is implicitly terminated with a NULL
character, @i{i.e.} a 0 byte.

The Poke standard library provides a couple of utility functions to
convert between string values and character arrays: @code{catos} and
@code{stoca}.

@code{catos} gets an array of characters and returns an equivalent
string.  For example:

@example
(poke) catos (['a','b','c'])
"abc"
@end example

@noindent
@code{stoca} gets a string and an array and sets the element of the
array to the characters composing the string.  For example:

@example
(poke) var a = char[3]()
(poke) stoca ("abc", a)
(poke) a
[0x61UB,0x62UB,0x63UB]
@end example

@node Structuring Data
@chapter Structuring Data

In the previous chapter we learned how to manipulate pre-defined
entities like bytes, integers and strings.  One of the big points of
poke, however, is that it allows you to define your own data
abstractions, and to operate in term of them.  This is achieved by
defining data structures.

@menu
* The SBM Format::              The Stupid BitMap Format
* Poking a SBM Image::	        A need for abstraction.
* Modifying SBM Images::        Modifying existing data.
* Defining Types::	        Abstracting data structures.
* Pickles::		        Pickling useful abstractions.
* Poking Structs::	        Abstracting heterogeneous data.
* How Structs are Built::       How poke builds struct values.
* Variables in Structs::
* Functions in Structs::
* Struct Methods::              Operations provided by built structs.
* Padding and Alignment::       Reserved fields, payloads and more.
* Dealing with Alternatives::   Conditional decoding.
* Structured Integers::         Structs that are stored like integers.
* Working with Incorrect Data:: Non-strict mapping to the rescue.
@end menu

@node The SBM Format
@section The SBM Format

@subsection Images as Maps of Pixels

There are two main ways to store two-dimensional images in computers.

One is to explicitly store the different @dfn{pixels} that compose the
image.  In these @dfn{bitmaps}, the pixels are arranged sequentially
and implicitly organized into @dfn{lines}.  A header typically
provides information to determine how many pixels fit in each line:

@example
 <--- line_width  --->
| pixel | pixel | @dots{} | pixel | pixel | @dots{} | @dots{}
|        line 1       |         line 2      |
@end example

Several properties have to be encoded for each pixel, depending on the
sophistication of the image: for monochrome images each pixel can be
just either switched on or off, so a single bit could be used to
encode each pixel (this is the origin of the term ``bitmap'').  When
color is added, a bit is no longer sufficient: the color of the pixel
shall be encoded in some way, typically using a color schema such as
RGB, that requires triplet of little integers to be encoded.  If
transparency is to be supported, the degree of transparency of the
pixel shall also be encoded somehow.

The other way to store an image is to store a functional description
of the ``painted'' parts of the image.  This functional description
usually contains instructions like ``paint a line of thickness 1 and
color red from the coordinate 10,20 to coordinate 10,40''.  Once
executed with certain parameters (like the desired resolution) the
functional description generates a bitmap.  Image formats using this
approach are commonly known as @dfn{vectorial formats}.

When it comes to bitmaps, there are a plethora of different formats
out there (bmp, jpg, png) competing in terms of capabilities such as
higher color depths, better resolution, support for transparency
(alpha channels), higher compression level, and the like.  These
capabilities greatly complicate these formats, but ultimately any
bitmap can be reduced to a sequence of pixels, which can be further
structured in terms of @dfn{lines}.

We don't know enough poke yet to handle the complications of these
real-life bitmap formats, so in subsequent sections we introduce a
very simple format for bitmaps, the Stupid BitMap format, or simply
SBM, that we will use for learning purposes.

But please do not feel disappointed: later in this book, when we
become more proficient pokers, we will be messing with these shiny
complex formats as well :)

@subsection SBM header

A SBM file starts with a header that contains a ``magic number''
composed of the three bytes 'S' (0x53UB), 'B' (0x42UB) and 'M'
(0x4dUB).  The next two bytes indicate the number of pixels per line,
and the number of lines, respectively.  In summary:

@example
                SBM header
+-------+-------+-------+-------+-------+
|  'S'  |  'B'  |  'M'  |  ppl  | lines |
+-------+-------+-------+-------+-------+
  byte0   byte1   byte2   byte3   byte4
@end example

@noindent
Here @code{ppl} stands for pixels-per-line.  The encoding of these
fields implies that the bigger image that can be encoded in SBM has
dimensions 255x255.

@subsection SBM data

Albeit stupid, SBM is a colorful format.  It supports more than a
million colors, encoding each color in what is known as @dfn{RGB24}.
In RGB24, each color is encoded using three little integers,
specifying the amount of red, green and blue that, added together,
compose the color.

The name RGB24 comes from the fact that each color is encoded using
three bytes, or 24 bits.  Therefore, each pixel in a SBM image is
encoded using three consecutive bytes:

@example
        SBM pixel
+-------+-------+-------+
|  Red  | Green | Blue  |
+-------+-------+-------+
  byte0   byte1   byte2
@end example

@noindent
Each byte in this encoding determines the amount of the base color
(red, green or blue) that compose the color.  We will talk more about
these components later.

@node Poking a SBM Image
@section Poking a SBM Image

@subsection P is for poke

Let's compose our first SBM image, using poke.  The image we want to
encode is the very simple rendering of the letter @code{P} shown in
the figure below.

@example
  | 0 | 1 | 2 | 3 | 4 |
  +---+---+---+---+---+
0 |   | * | * |   |   |
1 |   | * |   | * |   |
2 |   | * |   | * |   |
3 |   | * | * |   |   |
4 |   | * |   |   |   |
5 |   | * |   |   |   |
6 |   | * |   |   |   |
@end example

@noindent
The image has seven lines, and there are five pixels per line,
@i{i.e.} the dimension of the image in pixels is 5x7.  Also, the pixels
denoted by asterisks are red, whereas the pixels denoted with empty
spaces are white.  In other words, our image uses a red foreground
over a white background.  The ``painted'' pixels are called foreground
pixels, the non painted pixels are called background pixels.

@subsection Preparing the Canvas

The first thing we need is some suitable IO space where to encode the
image.  Let's fire up poke and create a memory buffer:

@example
$ poke
[@dots{}]
(poke) .mem image
The current IOS is now `*image*'.
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

@noindent
Freshly created memory IO spaces are 4096 bytes long, and that's big
enough for our little image.  If we wanted to work with more data,
remember that memory IO spaces will grow automagically when poked past
their size.

@subsection Poking the Header

The first three bytes of the header of a SBM file contains the magic
number that identifies the file as a SBM bitmap.  We can poke these
bytes very easily:

@example
(poke) byte @@ 0#B = 'S'
(poke) byte @@ 1#B = 'B'
(poke) byte @@ 2#B = 'M'
@end example

@noindent
The next couple of bytes encode the dimensions of the bitmap, in this
case 5x7:

@example
(poke) byte @@ 3#B = 5
(poke) byte @@ 4#B = 7
@end example

@cindex truncation
@cindex conversions
@cindex coercions
@noindent
There is something worth noting in this last mapping.  Even though we
were poking bytes (passing the @code{byte} type specifier to the map
operators) we specified the 32-bit signed integers @code{5} and
@code{7} instead of @code{5UB} and @code{7UB}.  When poke finds a
situation like this, where certain kind of integers are expected but
other kind are provided, it converts the value from the provided type
to the expected type.  This conversion may result in truncation (think
about converting, say 0xfff to an unsigned byte, whose maximum
possible value is 0xff) but certainly not in the case at hands.

The final header looks like:

@example
(poke) dump :size 16#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 0700 0000 0000 0000 0000 0000  SBM.............
@end example

@subsection Poking the Pixels

Now that we have written a SBM header, we have to encode the sequence
of pixels composing the image.

Recall that every pixel is encoded using three bytes, that conform a
RGB24 color.  We have two kinds of pixels in our image: white pixels,
and red pixels.  In RGB24 white is encoded as @code{(255,255,255)}.
Pure red is encoded as @code{(255,0,0)}, but to make things more
interesting we will be using a nicer tomato-like red
@code{(255,99,71)}.

Therefore, poking a white pixel at some offset @var{offset} would
involve the following operations:

@example
(poke) byte @@ @var{offset} = 255
(poke) byte @@ @var{offset}+1#B = 255
(poke) byte @@ @var{offset}+2#B = 255
@end example

@noindent
Likewise, the operations to poke a tomato pixel would look like:

@example
(poke) byte @@ @var{offset} = 255
(poke) byte @@ @var{offset}+1#B = 99
(poke) byte @@ @var{offset}+2#B = 71
@end example

@noindent
To ease things a bit, we can define variables with the color
components for both foreground and background pixels:

@example
(poke) var bg1 = 255
(poke) var bg2 = 255
(poke) var bg3 = 255
(poke) var fg1 = 255
(poke) var fg2 = 99
(poke) var fg3 = 71
@end example

@noindent
Then to poke a foreground pixel would involve doing:

@example
(poke) byte @@ @var{offset} = fg1
(poke) byte @@ @var{offset}+1#B = fg2
(poke) byte @@ @var{offset}+2#B = fg3
@end example

At this point, you may feel that the perspective of mapping the pixels
of our image is not very appealing, considering we have 5x7 = 35
pixels in our image. We will need to poke 35 * 3 = 105 bytes. We may
feel tempted to, somehow, use a bigger integer to ``encapsulate'' the
bytes.  Using the bit-concatenation operator, we could do something
like:

@example
(poke) var bg = 255UB::255UB::255UB
(poke) var fg = 255UB::99UB::71UB
(poke) bg
(uint<24>) 0xffffff
(poke) fg
(uint<24>) 0xff6347
@end example

@noindent
This encodes each color with a 24-bit unsigned integer. When looking
at the hexadecimal values of @code{bg} and @code{fg} above, note that
0xff = 255, 0x63 = 99 and 0x47 = 71.  Each byte seems to be in the
right position in the 24-bit containing number.  Now, poking a pixel
at some given offset should be as easy as issuing just
one map operation, right?  Let's see, using some arbitrary offset 10#B:

@example
(poke) uint<24> @@ 10#B = fg
(poke) dump :from 10#B :size 4#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
0000000a: 4763 ff00                                Gc..
@end example

@noindent
If your current endianness is little (@i{i.e.} you are running on a x86
system or similar) you will get the dump above.  The bytes are
reversed, and consequently the resulting pixel has the wrong color.
Our little trick didn't work :(

So are we doomed to poke three bytes for each pixel we want to poke in
our image?  No, not really.  The Poke language provides a construction
oriented to alleviate cases like this, where several similar elements
are to be ``encapsulated'' in a container.  These constructions are
called @dfn{arrays}.

Using array values, we can define the foreground and background colors
like this:

@example
(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]
@end example

@noindent
All the elements on an array should be of the same kind, @i{i.e.} of the
same type.  Therefore, this is not allowed:

@example
(poke) [1,"foo"]
<stdin>:1:1: error: array initializers should be of the same type
[1,"foo"];
^~~~~~~~~
@end example

Given an array value, it is possible to query for the number of values
contained in it (called @dfn{elements}) by using the @code{'length}
value attribute.  For example:

@example
(poke) bga'length
3UL
@end example

@noindent
Tells us that the array value stored in the variable @code{bga} has
three elements.

How can we poke an array value?  We know that the map operator accepts
two operands: a type specifier and the value to map.  The type
specifier of an array of three bytes is denoted as @code{byte[3]}.
Therefore, we can again try to poke a foreground pixel at offset 10#B,
this time using @code{fga}:

@example
(poke) byte[3] @@ 10#B = fga
(poke) dump :from 10#B :size 4#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
0000000a: ff63 4700                                 .cG.
@end example

@indent
This time, the bytes were written in the right order.  This is because
array elements are always written using their ``written'' ordering,
with no mind to endianness.  We can also map a pixel from a given
offset:

@example
(poke) byte[3] @@ 10#B
[255UB,99UB,71UB]
@end example

@subsection Poking Lines

At this point, we could encode the 40 pixels composing the image, by
issuing the same number of pokes of @code{byte[3]} arrays.  However,
we can simplify the task even further.

Our pixels are arrays of bytes, denoted by the type specifier
@code{byte[3]}.  Similarly, we could conceive arrays of 32-bit signed
integers, denoted by @code{int[3]}, or arrays of bits, denoted by
@code{uint<1>[3]}.  But, is it possible to have arrays of other
arrays?  Yes, it is:

@example
(poke) [[1,2],[3,4]]
@end example

@noindent
The value above is an array of two arrays of two integers each.  If we
wanted to map such an array, what would be the type specifier we would
need to use?  It would be @code{int[2][2]}, which should be read from
right-to-left as ``array of two arrays of two integers''.  Let's map
one from an arbitrary offset in our IO space:

@example
(poke) int[2][2] @@ 100#B
[[0,0],[0,0]]
@end example

Consider again the sequence of pixels composing the image.  Using the
information we have in the SBM header, we can group the pixels in the
sequence into ``lines''.  In our example image, each line contains 5
pixels.  It would be natural to express each line as a sequence of
pixels.  The first line in our image would be:

@example
(poke) var l0 = [bga,fga,fga,bga,bga]
(poke) l0
[[255UB,255UB,255UB],[255UB,99UB,71UB],@dots{}]
@end example

@noindent
Let's complete the image lines:

@example
(poke) var l0 = [bga,fga,bga,fga,bga]
(poke) var l1 = [bga,fga,bga,fga,bga]
(poke) var l2 = [bga,fga,fga,bga,bga]
(poke) var l3 = [bga,fga,bga,bga,bga]
(poke) var l4 = l3
(poke) var l5 = l4
@end example

@noindent
Note how we exploited the fact that the three last lines of our image
are identical, to avoid to write the same array thrice.  Array values
can be assigned, and in general manipulated, like any other kind of
value, such as integers or strings.

At this point, we could poke the pixels line-by-line.  What would be
the type specifier for a line?   A line is an array of five arrays of
3 bytes each, so the type specifier would be @code{byte[3][5]}.  Let's
do that:

@example
(poke) byte[3][5] @@ 5#B = l0
(poke) byte[3][5] @@ 10#B = l1
(poke) byte[3][5] @@ 15#B = l2
(poke) byte[3][5] @@ 20#B = l3
(poke) byte[3][5] @@ 25#B = l4
(poke) byte[3][5] @@ 30#B = l5
(poke) byte[3][5] @@ 35#B = l6
@end example

@noindent
Not bad, we went from poking 105 bytes in the IO space to poking six
lines.  But we can still do better@dots{}

@subsection Poking Images

When we poked the lines at the end of the previous section, we had to
increase the offset in every map operation.  This is inconvenient.

In the same way that a sequence of bytes can be abstracted in a line,
a sequence of lines can be abstracted in an image.  It follows that we
can look at the image data as an array of lines.  But lines are
themselves arrays of arrays@dots{} no matter, there is no limit on the
number of arrays-of levels that you can nest.

So, let's define our image as an array of the lines defined above:

@example
(poke) var image_data = [l0,l1,l2,l3,l4,l5]
(poke) image_data
[[[255UB,255UB,255UB],[255UB,99UB,71UB],[255UB,99UB,71UB]@dots{}]@dots{}]
@end example

@noindent
What would be the type specifier for an image?  It would be an array
of seven arrays of five arrays of three bytes each, in other words
@code{byte[3][5][7]}.  Let's poke the pixels:

@example
(poke) byte[3][5][6] @@ 5#B = image_data
@end example

This is an example of how abstraction can simplify the handling of
binary data: we switched from manipulating bytes to manipulate higher
abstractions such as colors, lines and images.  We achieved that by
structuring the data in a way that reflects these abstractions.
That's the way of the Poker.

@subsection Saving the Image

Now that we have completed the SBM image in our buffer @code{*image*},
it is time to save it to disk.  For that, we can use the
@command{save} command we are already familiar with.

We know that the SBM image starts at offset 0#B, but what is the size
of its entire binary representation?  The header is easy: it spans for
5 bytes.  The size of the sequence of pixels can be derived from the
pixels per line byte, and the number of lines byte.  We know that each
pixel occupies 3 bytes, so calculating@dots{}

@example
(poke) var ppl = byte @@ 3#B
(poke) var lines = byte @@ 4#B
(poke) save :from 0#B :size 5#B + ppl#B * lines#B :file "p.sbm"
@end example

@noindent
Note how we expressed ``ppl bytes'' as @code{ppl#B}, and ``lines
bytes'' as @code{lines#B}.  This is the same than expressing ``10
bytes'' as @code{10#B}.  We will talk more about these united values
later.

There is another way of getting the size of the stream of pixels.
Recall that we have the entire set of pixels, structured as lines,
stored in the variable @code{image_data}.  Given an array, it is
possible to query for its size using the @code{'size} @dfn{attribute}:

@example
(poke) .set obase 10
(poke) [1,2,3]'size
96UL#b
@end example

@noindent
The above indicates that the size of the array of the three integers
1, 2 and 3 is 96 bits.  Using that attribute, we can also obtain the
size of the pixels in the image:

@example
(poke) image_data'size
720UL#b
@end example

@noindent
And we can use it in the save command:

@example
(poke) save :from 0#B :size 5#B + image_data'size :file "p.sbm"
@end example

Using either strategy, at this point a file named @file{p.sbm} should
have been written in the current working directory, containing our ``P
is for poke'' image.  Keep that file around, because we will be poking
it further!

@node Modifying SBM Images
@section Modifying SBM Images

@subsection Reading a SBM File

Let's open with poke the cute image we created in the last section,
@file{p.sbm}:

@example
$ poke p.sbm
[@dots{}]
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff ffff ffff ffff ffff  .....cG.........
00000050: ffff ffff 6347 ffff ffff ffff ffff ffff  ....cG..........
00000060: ffff ff63 47ff ffff ffff ffff ffff       ...cG.........
@end example

@noindent
You can see the @code{P} in the ASCII column, right? If it wasn't for
the header, it would be pictured almost straight. This is because dump
shows 16 bytes per row, and our image has lines that are 15 bytes
long.  This is a happy coincidence: you definitely shouldn't expect to
see ASCII art in the dump output of SBM files in general! :)

Now let's read the image's metadata from the header: pixels per line
and how many lines are contained in the image:

@example
(poke) var ppl = byte @@ 3#B
(poke) ppl
5UB
(poke) var lines = byte @@ 4#B
(poke) lines
7UB
@end example

@noindent
All right, our image is 7x5.  Knowing that each pixel occupies three
bytes, and that each line contains @code{ppl} pixels, and that we have
@code{lines} lines, we can map the entire image data using an array
type specifier:

@example
(poke) var image_data = byte[3][ppl][lines] @@ 5#B
(poke) image_data
[[[255UB,255UB,255UB],[255UB,99UB,71UB], @dots{}]@dots{}]
@end example

@subsection Painting Pixels

The ``P is for poke'' slogan was so successful and widely appraised
that the recutils@footnote{http://www.gnu.org/s/recutils} chaps wanted
to do a similar campaign ``R is for recutils''.  For that purpose,
they asked us for a tomato-colored SBM image with an @code{R} in it.

Our creative department got at it, and after a lot of work they came
with the following design:

@example
  | 0 | 1 | 2 | 3 | 4 |
  +---+---+---+---+---+
0 |   | * | * |   |   |
1 |   | * |   | * |   |
2 |   | * |   | * |   |
3 |   | * | * |   |   |
4 |   | * | * |   |   |
5 |   | * |   | * |   |
6 |   | * |   | * |   |
@end example

@noindent
Observe that this design really looks like our @code{P} (so much for a
creative department).  The bitmap has exactly the same dimensions, and
difference are just three pixels, that pass from being background
pixels to foreground pixels.

Therefore, it makes sense to read our @file{p.sbm} and use it as a
base, completing the missing pixels.  We saw in the last section how
to read a SBM image.  This time, however, we will copy the image first
to a memory IO space to avoid overwriting @file{p.sbm}:

@example
$ poke p.sbm
[@dots{}]
(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	MEMORY		0x00000000#B	0x00001000#B	*scratch*
  #0	FILE	rw	0x00000000#B	0x0000006e#B	./p.sbm
(poke) copy :from_ios 0 :from 0#B :to 0#B :size iosize (0)
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff ffff ffff ffff ffff  .....cG.........
00000050: ffff ffff 6347 ffff ffff ffff ffff ffff  ....cG..........
00000060: ffff ff63 47ff ffff ffff ffff ffff 0000  ...cG...........
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

@noindent
Good.  Now let's map the contents of the image, both header information
and the sequence of pixels:

@example
(poke) var ppl = byte @@ 3#B
(poke) var lines = byte @@ 4#B
(poke) var image_data = byte[3][ppl][lines] @@ 5#B
@end example

@noindent
Let's modify the image.  Since the dimensions of the new image are
exactly the same, the header remains the same.  It is the pixel
sequence that is different.  We basically need to turn the pixels at
coordinates @code{(4,2)}, @code{(5,3)} and @code{(6,3)} from
background pixels to foreground pixels.

@cindex mapped values
Remember how we would change the value of some integer in the IO
space?  First, we would map it into a variable, change the value, and
then poke it back to the IO space.  Something like this:

@example
(poke) var n = int @@ @var{offset}
(poke) n = n + 1
(poke) int @@ @var{offset} = n
@end example

@noindent
This three-steps process is necessary because in the @code{n = n + 1}
above we are modifying the value of the variable @code{n}, not the
integer actually stored at offset @var{offset} in the current IO
space.  Therefore we have to explicitly poke it back if we want the IO
space to be updated as well.

Array values (and, as we shall see, other ``composited'' values) are
different: when they are the result of the application of the map
operator, the resulting values are @dfn{mapped} themselves.

When a Poke value is mapped, updating their elements have a side
effect: the area corresponding to the updated element, in whatever IO
space it is mapped on, is updated as well!

Why is this?  The map operator, regardless of the kind of value it is
mapping, always returns a @emph{copy} of the value found stored in the
IO space.  We already saw how this worked with integers.  However, in
Poke values are copied around using a mechanism called ``shared
value''.  This means that when a composite value like an array is
copied, its elements are shared by both the original value and the new
value.

We can depict this graphically for better understanding.  A Poke array
like:

@example
(poke) var a = [1,2,3]
@end example

@noindent
is stored in memory like this:

@example
   +---+---+---+
 a | 1 | 2 | 3 |
   +---+---+---+
@end example

If we make a copy of the array in another variable @code{b}:

@example
(poke) var b = a
@end example

@noindent
we get

@example
   +---+---+---+      +---+---+---+
 a | 1 | 2 | 3 |    b | 1 | 2 | 3 |
   +---+---+---+      +---+---+---+
@end example

@noindent
Note how each of the integer elements has been copied to the new
value.  The resulting two arrays can then be modified independently:

@example
(poke) a[1] = 5
@end example

@noindent
resulting in:

@example
   +---+---+---+      +---+---+---+
 a | 1 | 5 | 3 |    b | 1 | 2 | 3 |
   +---+---+---+      +---+---+---+
@end example

However, consider the following array whose elements are also arrays:

@example
(poke) var a = [[1,2],[3,4],[5,6]]
@end example

@noindent
This array is stored in memory like this:

@example
   +---+---+---+
 a |   |   |   |
   +---+---+---+
     |   |   |   +---+---+
     |   |   +---| 5 | 6 |
     |   |       +---+---+
     |   |
     |   |       +---+---+
     |   +-------| 3 | 4 |
     |           +---+---+
     |
     |           +---+---+
     +-----------| 1 | 2 |
                 +---+---+
@end example

If now we make a copy of the same array into another variable
@code{b}:

@example
(poke) var b = a
@end example

@noindent
The elements of the array are copied ``by shared value'', @i{i.e.}

@example
   +---+---+---+           +---+---+---+
 a |   |   |   |         b |   |   |   |
   +---+---+---+           +---+---+---+
     |   |   |   +---+---+   |   |   |
     |   |   +---| 5 | 6 |---+   |   |
     |   |       +---+---+       |   |
     |   |                       |   |
     |   |       +---+---+       |   |
     |   +-------| 3 | 4 |-------+   |
     |           +---+---+           |
     |                               |
     |           +---+---+           |
     +-----------| 1 | 2 |-----------+
                 +---+---+
@end example

@noindent
The elements are indeed shared between the original array and the
copy!  If now we modify any of the shared values, this will be
reflected in both containing values:

@example
(poke) a[1][1] = 9
(poke) a
[[1,2],[3,9],[5,6]]
(poke) b
[[1,2],[3,9],[5,6]]
@end example

@noindent
or graphically:

@example
   +---+---+---+           +---+---+---+
 a |   |   |   |         b |   |   |   |
   +---+---+---+           +---+---+---+
     |   |   |   +---+---+   |   |   |
     |   |   +---| 5 | 6 |---+   |   |
     |   |       +---+---+       |   |
     |   |                       |   |
     |   |       +---+---+       |   |
     |   +-------| 3 | 9 |-------+   |
     |           +---+---+           |
     |                               |
     |           +---+---+           |
     +-----------| 1 | 2 |-----------+
                 +---+---+
@end example

Back to our image, it follows that if we wanted to change the color of
some SBM pixel stored at offset @var{offset}, we would do this:

@example
(poke) var a = byte[3] @@ @var{offset}
(poke) a[1] = 10
@end example

@noindent
There is no need to poke the array back explicitly: the side effect of
assigning 10 to a[1] is that the byte at offset @code{@var{offset}+1}
is poked.

Generally speaking, mapped values can be handled in exactly the same
way than non-mapped values.  This is actually a very central concept
in poke.  However, it is possible to check whether a given value is
mapped or not using the @code{'mapped} attribute.

As we said, @dfn{simple values} such as integers and strings are never
mapped, regardless of where they come from.  Both @code{ppl} and
@code{lines} are integers, therefore:

@example
(poke) ppl'mapped
0
(poke) lines'mapped
0
@end example

@noindent
However, @code{image_data} is an array that was the result of the
application of a map operator, so:

@example
(poke) image_data'mapped
1
@end example

@noindent
When a value is mapped, you can ask for the offset where it is mapped,
and the IO space where it is mapped, using the attributes
@code{'offset} and @code{'ios} respectively.  Therefore:

@example
(poke) image_data'ios
1
(poke) image_data'offset
40UL#b
@end example

@noindent
In other words, @code{image_data} is mapped in the IO space with id 1
(the @code{*scratch*} buffer) at offset 40 bits, or 5 bytes.  We
already knew that, because we mapped the image data ourselves, but in
other situations these attributes are most useful.  We shall see how
later.

Well, at this point it should be clear how to paint pixels.  First,
let's define our background and foreground pixels:

@example
(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]
@end example

@noindent
Then, we just update the pixels in the image data using the right
coordinates:

@example
(poke) image_data[4][2] = fga
(poke) image_data[5][3] = fga
(poke) image_data[6][3] = fga
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff 6347 ffff ffff ffff  .....cG.cG......
00000050: ffff ffff 6347 ffff ffff 6347 ffff ffff  ....cG....cG....
00000060: ffff ff63 47ff ffff ff63 47ff ffff 0000  ...cG....cG.....
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
@end example

@subsection Cropping the R

Looking at our new image, we realize that the first and the last
column are all background pixels.  We are aware that the recutils
project is always short of resources, so we would like to modify the
image to remove these columns, @dfn{cropping} it so it looks like
this:

@example
  | 0 | 1 | 2 |
  +---+---+---+
0 | * | * |   |
1 | * |   | * |
2 | * |   | * |
3 | * | * |   |
4 | * | * |   |
5 | * |   | * |
6 | * |   | * |
@end example

@noindent
In order to perform this operation we need to rework the stream of
pixels to reflect the desired result, and then to update the header
metadata accordingly.

@subsection Shortening and Shifting Lines

Let's think in term of lines.  In the original image, each line has 5
pixels, that we can enumerate as:

@example
     +----+----+----+----+----+
line | p0 | p1 | p2 | p3 | p4 |
     +----+----+----+----+----+
@end example

@noindent
What we want is to crop out the first and the last column, so the
resulting line would look like:

@example
     +----+----+----+
line | p1 | p2 | p3 |
     +----+----+----+
@end example

Let's get the first line from the original @code{image_data}:

@example
(poke) var l0 = image_data[0]
(poke) l0
[[255UB,255UB,255UB],[255UB,99UB,71UB],@dots{}]
@end example

@noindent
We could create the corresponding cropped line, by doing something
like this:

@example
(poke) var cl0 = [l0[1],l0[2],l0[3]]
@end example

@cindex arrays, trimming
@noindent
And the same for the other lines.  However, Poke provides a better way
to easily obtain sub portions of arrays.  It is called @dfn{trimming}.
Given an array like the line @code{l0}, we can obtain the desired
portion of it by issuing:

@example
(poke) l0[1:4]
[[255UB,99UB,71UB],[255UB,99UB,71UB]]
@end example

@noindent
Note how the limits of the semi-open interval specified in the trim
reflect array indexes, and hence they are 0 based.  Also, the left
limit is closed and the right limit is open.  The result of an array
trimming is always another array, which may be empty:

@example
(poke) l0[1:3]
[[255UB,99UB,71UB]]
@end example

Armed with this new operation, we can very easily mount the sequence
of pixels for our cropped image:

@example
(poke) var l0 = image_data[0]
(poke) var l1 = image_data[1]
(poke) var l2 = image_data[2]
(poke) var l3 = image_data[3]
(poke) var l4 = image_data[4]
(poke) var l5 = image_data[5]
@end example

@noindent
And then update the lines in the mapped image data:

@example
(poke) image_data[0] = l0[1:4]
(poke) image_data[0] = l1[1:4]
(poke) image_data[1] = l2[1:4]
(poke) image_data[2] = l3[1:4]
(poke) image_data[3] = l4[1:4]
(poke) image_data[4] = l5[1:4]
@end example

@subsection Updating the Header

The last step is to update the header to reflect the new dimensions of
the image:

@example
(poke) byte[] @@ 0#B = ['S','B','M']
(poke) byte @@ 3#B = 3
(poke) byte @@ 4#B = 7
@end example

@noindent
And we are done.  Note how this time we wrote the magic bytes as an
array, to save some typing and silly manual offset arithmetic.  You
may have noticed that the type specifier we used this time in the map
is @code{byte[]} instead of @code{byte[3]}.  This type specifier
denotes an array of any number of bytes, which certainly includes
arrays of three bytes, like in the example.

@subsection Saving the Result

And finally, let's write out the new file as @file{r.sbm}:

@example
(poke) save :from 0#B :size 5#B + image_data'size :file "r.sbm"
@end example

@node Defining Types
@section Defining Types

@subsection Naming your Own Abstractions

During the process of creating and manipulating SBM files we soon
started talking about things like lines, pixel sequences, pixels,
colors, and so on.  What is more, very naturally we started
@emph{thinking} in terms of these entities: let's drop this or that
line, or let's change the green component of this pixel.

Consider for example RGB colors.  We know that each color is defined
by three levels of light: red, green and blue.  These components are
also called color beams.  Since each color beam has a range of 0 to
255, many formats, like SBM, use bytes to encode them.

Therefore, in the previous sections we used the type specifier
@code{byte} when we needed to map RGB color beams, like in:

@example
(poke) byte[3] @@ 5#B
@end example

@noindent
Recall that the mapping operation above means ``map three bytes at the
offset 5 bytes in the current IO space''.  But what we really want is
to map color beams, not bytes!

@cindex type
Poke provides a way to assign names to type specifiers:

@example
(poke) type RGB_Color_Beam = byte
@end example

@noindent
The definition above tells poke that a RGB color beam is composed of a
byte, @i{i.e.} an unsigned 8-bit integer.  Any type specifier can be used
at the right side of the assignment sign, and also names of already
defined types.  From this point on, we can map in terms of color
beams:

@example
(poke) RGB_Color_Beam[3] @@ 5#B
@end example

@noindent
Meaning ``map three RGB color beams at the offset 5 in the current IO
space''.

Once a type is defined, the name can be used anywhere where a type
specifier is expected.

By the way, we mentioned many times how @code{byte} is a synonym for
@code{uint<8>}, @code{int} is a synonym for @code{int<32>} and so on.
These synonyms are actually result of type definitions that are in the
poke @dfn{standard library}.  This library is loaded by poke at
startup time.

@subsection Abstracting the Structure of Entities

Since we didn't know better, during our work with SBM images we had to
remember how these entities were constructed from more simple entities
such as bits and bytes, every time we needed to map them, or to poke
them.  For example, if we wanted to map a pixel at some particular
offset, we had to issue the following command:

@example
(poke) var pixel = byte[3] @@ 5#B
@end example

@noindent
Now that we made poke aware of what a RGB color beam is, we can
rewrite the above as:

@example
(poke) var pixel = RGB_Color_Beam[3] @@ 5#B
@end example

@noindent
This is better, but still adoleces from a big problem: what if at some
point the SBM pixels get expanded to also have a transparency index,
stored in a fourth byte?  If that ever happens (and @emph{will} happen
later in this book) then we would need to remember it before issuing
commands like:

@example
(poke) var image_data = RGB_Color_Beam[4][ppl][lines] @@ 5#B
@end example

To avoid this problem, we define yet another type, this time
describing the structure of a SBM pixel:

@example
(poke) type SBM_Pixel = RGB_Color_Beam[3]
@end example

@noindent
And then we can define @code{image_data} as a table of SBM pixels,
instead of as a table of triplets of RGB color beams:

@example
(poke) var image_data = SBM_Pixel[ppl][lines] @@ 5#B
@end example

This later definition doesn't need to be changed if we change the
definition of what a SBM pixel is.  This is called @dfn{encapsulation}
and is a very useful abstraction in computing.

@node Pickles
@section Pickles

@subsection poke Commands versus Poke Constructions

In @ref{Nomenclature} we mentioned that poke, the program, implements
a domain specific programming language called Poke, with a big p.  In
the examples so far we have already used the Poke language, somewhat
extensively, while interacting with the program using the REPL.

For example, in:

@example
(poke) 10 + 2
12
@end example

@noindent
We are giving poke a @dfn{Poke expression} @code{10 + 2} to be
evaluated.  Once the expression is evaluated, the REPL prints the
resulting value for us.

Similarly, when we define a variable or a type with @code{var} and
@code{type} respectively, we are providing poke definitions to be
evaluated.  When we assign a value to a variable we are actually
providing a Poke statement to be executed.

So the REPL accepts @dfn{poke commands}, some of which happen to be
Poke expressions, definitions or statements.  But we also have used
dot-commands like @code{.file} or @code{.info}.  These dot-commands
are not part of the Poke programming language.

Every time we insert a line in the REPL and hit @key{enter}, poke
recognizes the nature of the line, and then does the right thing.  If
the line is recognized as a Poke expression, for example, the Poke
compiler is used to compile the statement into a routine, that is
executed by the Poke Virtual Machine.  The resulting value is then
printed for the benefit of the user.

@subsection Poke Files

Poke programs are basically a collection of definitions and
statements, which most often than not are stored in files, which
avoids the need to type them again and again.  By convention, we use
the @file{.pk} file extension when naming files containing Poke
programs.

Remember how we defined the foreground and background pixels for
@code{p.sbm} in the REPL?

@example
(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]
@end example

@noindent
Where @code{bga} is a white pixel and @code{fga} is a tomato colored
pixel.  We could write these definitions in a file @file{colors.pk}
like this:

@example
var white = [255UB, 255UB, 255UB];
var tomato = [255UB, 99UB, 71UB];
@end example

@noindent
Note that variable definitions in Poke are terminated by a semicolon
(@code{;}) character, but we didn't need to specify them when we
issued the definitions in the REPL.  This is because poke adds the
trailing semicolon for us when it detects a Poke construction
requiring it is introduced in the REPL.

@noindent
Another difference is that Poke constructions can span for multiple
lines, like in most programming languages.  For example, we could have
the following variable definition in a file:

@example
var matrix = [[10, 20, 30],
              [40, 50, 60],
              [70, 80, 90]];
@end example

@cindex loading files
@cindex .load
Once we have written our @file{colors.pk} file, how can we make poke
aware of it?  A possibility is to use the @code{load} construction:

@example
(poke) load colors
@end example

@noindent
Assuming a file named @file{colors.pk} exists in the current working
directory, poke will load it and evaluate its contents.  After this,
we can use the colors:

@example
(poke) tomato
[0xffUB,0x63UB,0x47UB]
@end example

@noindent
Before, we would need to define these colors every time we would like
to poke SBM files or, in fact, any RGB24 data.  Now we just have to
load the file @file{colors.pk} and use the variables defined there.

If you try to load a file whose name contains a dash character
(@code{-}) you will get an error message:

@example
(poke) load my-colors
<stdin>:1:8: error: syntax error, unexpected '-', expecting ';'
load my-colors;
       ^
@end example

@noindent
This is because the argument to @code{load} is interpreted as a Poke
identifier, and dash characters are not allowed in identifiers.  To
alleviate this problem, you can also specify the name of the file to
load encoded in a string, line in:

@example
(poke) load "my-colors.pk"
@end example

@noindent
If you use this form of @code{load}, however, you are required to
specify the complete name of the file, including the @file{.pk} file
extension.

Since loading files is such a common operation, poke provides a
dot-command @code{.load} that does auto-completion:

@example
(poke) .load my-colors.pk
@end example

@noindent
Which is equivalent to @code{load "my-colors.pk"}.

Since @code{load} is part of the Poke language, it can also be used in
Poke programs stored in files.  We will explore this later.

@subsection Pickling Abstractions

In the last section we defined a couple of RGB colors @code{white} and
@code{tomato} in a file called @file{colors.pk}.  If we keep adding
colors to the file, we may end with a nice collection of colors that
are available to us at any time, by just loading the file.

Since there are many ways to understand the notion of ``color'', and
also many ways to implement these many notions, it would be better to
be more precise and call our file @file{rgb24.pk}, since the notion of
color we are using is of that RGB24.  While are at it, let's also
rename the variables to reflect the fact they denote not just any kind
of colors, but RGB24 colors:

@example
var rgb24_white = [255UB, 255UB, 255UB];
var rgb24_tomato = [255UB, 99UB, 71UB];
@end example

@noindent
At this point, we can also add the definitions of a couple of types to
our @file{rgb24.pk}:

@example
type RGB_Color_Beam = byte;
type RGB24_Color = RGB_Color_Beam[3];

var rgb24_white = [255UB, 255UB, 255UB];
var rgb24_tomato = [255UB, 99UB, 71UB];
@end example

@noindent
Any time we want to manipulate RGB24 colors we can just load the file
@file{rgb24.pk} and use these types and variables.

In poke parlance we call files like the above, that contain
definitions of conceptually related entities, @dfn{pickles}.  Pickles
can be very simple, like the @file{rgb24.pk} sketched above, or fairly
complicated like @file{dwarf.pk}.

It is common for pickles to @code{load} other pickles.  For example,
if we were to write a SBM pickle, we would load the RGB24 pickle from
it:

@example
load rgb24;

[@dots{} SBM definitions @dots{}]
@end example

@noindent
This way, when we @command{load sbm} from the repl, the dependencies
get loaded as well.

GNU poke includes several already written pickles for commonly used
file formats, and other domains.  The @code{load} construction knows
where these pickles are installed, so in order to load the pickle to
manipulate ELF files, for example, all you have to do is to:

@example
(poke) load elf
@end example

@subsection Exploring Pickles

As we have seen, a pickle provides Poke variables, types and functions
related to some definite domain.  Let's say we are interested in
editing an ELF file.  We know that GNU poke comes with a pre-installed
pickle named @file{elf.pk}.  We can load it like this:

@example
(poke) load elf
@end example

By convention, the entities provided by a pickle @file{foo.pk} are
usually prefixed like:

@itemize @minus
@item
Variables, units and functions are prefixed with @code{foo_}.
@item
Types are prefixed with @code{Foo_}.
@end itemize

@noindent
Therefore, once the pickle is loaded we can use the @command{.info}
dot-command in order to get a glimpse of the functionality provided by
the pickle. @xref{info command}.  Example:

@example
(poke) .info variables elf
Name            Declared at
elf_stb_names   elf-common.pk:53
elf_stt_names   elf-common.pk:72
elf_stv_names   elf-common.pk:84
@end example

@noindent
The above tells us that the @file{elf.pk} pickle provides these three
variables.  Types are way more interesting:

@example
(poke) .info types Elf64
Name                 Declared at
Elf64_Ehdr           elf-64.pk:184
Elf64_Off            elf-64.pk:26
Elf64_SectionFlags   elf-64.pk:130
Elf64_Shdr           elf-64.pk:152
Elf64_RelInfo        elf-64.pk:36
Elf64_Chdr           elf-64.pk:85
Elf64_Rela           elf-64.pk:61
Elf64_Phdr           elf-64.pk:169
Elf64_Sxword         elf-64.pk:24
Elf64_Sym            elf-64.pk:71
Elf64_Rel            elf-64.pk:45
Elf64_File           elf-64.pk:207
Elf64_Addr           elf-64.pk:25
Elf64_Group          elf-64.pk:121
Elf64_Dyn            elf-64.pk:98
Elf64_Xword          elf-64.pk:23
@end example

@noindent
We immediately notice the @code{Elf64_File}.  Looks like what we would
need to map in order to access the entire ELF data in some given IO
space (like a file.)  We can ask for more details about it:

@example
(poke) .info type Elf64_File
Class:      struct
Name:       Elf64_File
Complete:   no
Fields:
  Name     Type
  ehdr     Elf64_Ehdr
  shdr     Elf64_Shdr[]
  phdr     Elf64_Phdr[]
Methods:
  Name                     Type
  get_section_name         (offset<Elf_Word,8>)string
  get_symbol_name          (Elf64_Shdr,,offset<Elf_Word,8>)string
  get_sections_by_name     (string)Elf64_Shdr[]
  get_sections_by_type     (Elf_Word)Elf64_Shdr[]
  section_name_p           (string)int
  get_string               (offset<Elf_Word,8>)string
  get_group_signature      (Elf64_Shdr)string
  get_group_signatures     ()string[]
  get_section_group        (string)Elf64_Shdr[]
@end example

@noindent
We see that an @code{Elf64_File} has three fields, and the nature of
these files: a header @code{ehdr}, an array of section headers
@code{shdr} and an array of program headers @code{phdr}.

The type description above also lists the methods defined in the type.
Looking at the method name and its type is very usually revealing
enough to use it.  For example, we see that
@code{get_sections_by_name} returns an array of section headers,
characterized by some given section name:

@example
(poke) var elf = Elf64_File @@ 0#B
(poke) elf.get_sections_by_name (".text")
[Elf64_Shdr @{
  sh_name=0x1bU#B,
  sh_type=0x1U,
  sh_flags=#<ALLOC,EXECINSTR>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0xbUL#B,
  sh_link=0x0U,
  sh_info=0x0U,
  sh_addralign=0x1UL,
  sh_entsize=0x0UL#b
@}]
@end example

Ultimately, of course the best way to see what a pickle provides is to
go read its source code.

@subsection Startup

When poke starts it loads the file @file{.pokerc} located in our home
directory, if it exists.  This initialization file contains poke
commands, one per line.

If we wanted to get some Poke file loaded at startup, we could do it
by adding a load command to our @code{.pokerc}.  For example:

@example
# My poke configuration - jemarch
[@dots{}]
.load ~/.poke.d/mydefs.pk
@end example

@node Poking Structs
@section Poking Structs

@subsection Heterogeneous Related Data

Let's recap the structure of the header of a Stupid BitMap:

@example
                SBM header
+-------+-------+-------+-------+-------+
|  'S'  |  'B'  |  'M'  |  ppl  | lines |
+-------+-------+-------+-------+-------+
  byte0   byte1   byte2   byte3   byte4
@end example

@noindent
The header is composed of five fields, which actually compose three
different logical @dfn{fields}: a magic number, the number of pixels
per line, and the number of lines.

We could of course abstract the header using an array of five bytes,
like this:

@example
type SBM_Header = byte[5];
@end example

However, this would not capture the properties of the fields
themselves, which would need to be remembered by the user: which of
these five bytes correspond to the magic number?  Is the pixels per
line number signed or unsigned? @i{etc}.

Poke provides a much better way to abstract collections of
heterogeneous data: @dfn{struct types}.  Using a struct type we can
abstract the SBM header like this:

@example
type SBM_Header =
 struct
 @{
   byte[3] magic;
   uint<8> ppl;
   uint<8> lines;
 @}
@end example

Note how the struct has three named fields: @code{magic}, @code{ppl}
and @code{lines}.  @code{magic} is an array of three bytes, while
@code{ppl} and @code{lines} are both unsigned integers.

@subsection Mapping Structs

Once defined, struct types can be referred by name.  For example, we
can map the SBM header at the beginning of our file @file{p.sbm} like
this:

@example
(poke) SBM_Header @@ 0#B
SBM_Header @{
  magic=[0x53UB,0x42UB,0x4dUB],
  ppl=0x5UB,
  lines=0x7UB
@}
@end example

@noindent
The value resulting from the mapping is a struct value.  The fields of
struct values are accessed using the familiar dot-notation:

@example
(poke) var header = SBM_Header @@ 0#B
(poke) header.ppl * header.lines
35UB
@end example

@noindent
The total number of pixels in the image is 35.  Note how both
@code{header.ppl} and @code{header.lines} are indeed unsigned byte
values, and thus the result of the multiplication is also an unsigned
byte.  This could be problematic if the image contained more than 255
pixels, but this can be prevented by using a cast:

@example
(poke) header.ppl as uint * header.lines
35U
@end example

@noindent
Now the second operand @code{header.lines} is promoted to a 32-bit
unsigned value before the multiplication is performed.  Consequently,
the result of the operation is also 32-bit wide (note the suffix of
the result.)

@subsection Modifying Mapped Structs

Remember when we wanted to crop a SBM image by removing the first and
last row?  We updated the header in a byte by byte manner, like this:

@example
(poke) byte @@ 3#B = 3
(poke) byte @@ 4#B = 7
@end example

@noindent
Now that we have the header mapped in a variable, updating it is much
more easy and convenient.  The dot-notation is used to update the
contents of a struct field, by placing it at the left hand side of an
assignment:

@example
(poke) header.ppl = 3
(poke) header.lines = 7
@end example

@noindent
This updates the pixel per line and the number of lines, in the IO
space:

@example
(poke) dump :size 5#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d03 07                             SBM..
@end example

@node How Structs are Built
@section How Structs are Built

First we need to define some structure to use as an example.  Let's
say we are interesting in poking Packets, as defined by the Packet
Specification 1.2 published by the Packet Foundation (none less).

In a nutshell, each Packet starts with a byte whose value is always
@code{0xab}, followed by a byte that defines the size of the payload.
A stream of bytes conforming the payload follows, themselves followed
by another stream of the same number of bytes with ``control'' values.

We could translate this description into the following Poke struct
type definition:

@example
type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;
    byte[size] payload;
    byte[size] control;
  @};
@end example

There are some details described by the fictitious Packet
Specification 1.2 that are not covered in this simple definition, but
we will be attending to that later in this manual.

So, given the definition of a struct type like Packet, there are only
two ways to build a struct value in Poke.

One is to map it from some IO space.  This is achieved using the map
operator:

@example
(poke) Packet @@ 12#B
Packet @{
  magic == 0xab,
  size = 2,
  payload = [0x12UB,0x30UB],
  control = [0x1UB,0x1UB]
@}
@end example

The expression above maps a Packet starting at offset 12 bytes, in the
current IO space.  See the Poke manual for more details on using the
map operator.

The second way to build a struct value is to _construct_ one,
specifying the value to some, all or none of its fields.  It looks
like this:

@example
(poke) Packet @{size = 2, payload = [1UB,2UB]@}
Packet @{
  magic == 0xab,
  size = 2,
  payload = [0x1UB,0x2UB],
  control = [0x0UB,0x0UB]
@}
@end example

In either case, building a struct involves to determine the value of
all the fields of the struct, one by one.  The order in which the
struct fields are built is determined by the order of appearance of
the fields in the type description.

In our example, the value of @code{magic} is determined first, then
@code{size}, @code{payload} and finally @code{control}.  This is the
reason why we can refer to the values of previous fields when defining
fields, such as in the size of the @code{payload} array above, but not
the other way around: by the time @code{payload} is mapped or
constructed, the value of @code{size}, has already been mapped or
constructed.

What happens behind the curtains is that when poke finds the
definition of a struct type, like Packet, it compiles two functions
from it: a mapper function, and a constructor function.  The mapper
function gets as arguments the IO space and the offset from which to
map the struct value, whereas the constructor function gets the
template specifying the initial values for some, or all of the fields;
reasonable default values (like zeroes) are used for fields for which
no initial values have been specified.

These functions, mapper and constructor, are invoked to create fresh
values when a map operator @@ or a struct constructor is used in a Poke
program, or at the poke prompt.

@node Variables in Structs
@section Variables in Structs

Fields are not the only entity that can appear in the definition of a
struct type.

Suppose that after reading more carefully the Packet Specification 1.2
(that spans for several thousand of pages) we realize that the field
@code{size} doesn't really store the number of bytes of the payload
and control arrays, like we thought initially.  Or not exactly: the
Packet Foundation says that if @code{size} has the special value 0xff,
then the size is zero.

We could of course do something like this:

@example
type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    byte[size == 0xff ? 0 : size] payload;
    byte[size == 0xff ? 0 : size] control;
  @};
@end example

However, we can avoid replicating code by using a variable instead:

@example
type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;
  @};
@end example

Note how the variable can be used after it gets defined.  In the
underlying process of mapping or constructing the struct, the variable
is incorporated into the lexical environment.  Once defined, it can be
used in constraint expressions, array sizes, and the like.  We will
see more about this later.

Incidentally, it is of course possible to use global variables as
well.  For example:

@example
var packet_special = 0xff;
type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    var real_size = (size == packet_special ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;
  @};
@end example

In this case, the global @code{packet_special} gets captured in the
lexical environment of the struct type (in reality in the lexical
environment of the implicitly created mapper and constructor
functions) in a way that if you later modify @code{packet_special} the
new value will be used when mapping/constructing @emph{new} values of
type Packet.  Which is really cool, but let's not get distracted from
the main topic... :)

@node Functions in Structs
@section Functions in Structs

Further reading of the Packet Specification 1.2 reveals that each
Packet has an additional @code{crc} field.  The content of this field
is derived from both the payload bytes and the control bytes.

But this is no vulgar CRC we are talking about.  On the contrary, it
is a special function developed by the CRC Foundation in partnership
with the Packet Foundation, called superCRC (patented, TM).

Fortunately, the CRC Foundation distributes a pickle
@code{supercrc.pk}, that provides a @code{calculate_crc} function with
the following spec:

@example
fun calculate_crc = (byte[] data, byte[] control) int:
@end example

So let's use the function like this in our type, after loading the
supercrc pickle:

@example
load supercrc;

type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    int crc = calculate_crc (payload, control);
  @};
@end example

However, there is a caveat: it happens that the calculation of the CRC
may involve arithmetic and division, so the CRC Foundation warns us
that the @code{calculate_crc} function may raise E_div_by_zero.
However, the Packet 1.2 Specification tells us that in these
situations, the @code{crc} field of the packet should contain zero.
If we used the type above, any exception raised by
@code{calculate_crc} would be propagated by the mapper/constructor:

@example
(poke) Packet @@ 12#B
unhandled division by zero exception
@end example

A solution is to use a function that takes care of the extra needed
logic, wrapping calculate_crc:

@example
load supercrc;

type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    fun corrected_crc = int:
    @{
      try return calculate_crc (payload, control);
      catch if E_div_by_zero @{ return 0; @}
    @}

    int crc == corrected_crc;
  @};
@end example

Again, note how the function is accessible after its definition.  Note
as well how both fields and variables and other functions can be used
in the function body.  There is no difference to define variables and
functions in struct types than to define them inside other functions
or in the top-level environment: the same lexical rules apply.

@node Struct Methods
@section Struct Methods

At this point you may be thinking something on the line of ``hey,
since variables and functions are also members of the struct, I should
be able to access them the same way than fields, right?''.

So you will want to do:

@example
(poke) var p = Packet @ 12#B
(poke) p.real_size
(poke) p.corrected_crc
@end example

@noindent
But sorry, this won't work.

To understand why, think about the struct building process we sketched
above.  The mapper and constructor functions are derived/compiled from
the struct type.  You can imagine them to have prototypes like:

@example
Packet_mapper (IOspace, offset) -> Packet value
Packet_constructor (template)   -> Packet value
@end example

You can also picture the fields, variables and functions in the struct
type specification as being defined inside the bodies of Packet_mapper
and Packet_constructor, as their contents get mapped/constructed.  For
example, let's see what the mapper does:

@example
Packet_mapper:

  . Map a byte, put it in a local `magic'.
  . Map a byte, put it in a local `size'.
  . Calculate the real size, put it in a local `real_size'.
  . Map an array of real_size bytes, put it in a local `payload'.
  . Map an array of real_size bytes, put it in a local `control'.
  . Compile a function, put it in a local `corrected_crc'.
  . map a byte, call the function in the local `corrected_crc',
    complain if the values are not the same, otherwise put the
    mapped byte in a local `crc'.
  . Build a struct value with the values from the locals `magic',
    `size', `payload', `control' and `crc', and return it.
@end example

The pseudo-code for the constructor would be almost identical.  Just
replace "map a byte" with ``construct a byte''.

So you see, both the values for the mapped fields and the values for
the variables and functions defined inside the struct type end as
locals of the mapping process, but only the values of the fields are
actually put in the struct value that is returned in the last step.

This is where methods come in the picture.  A method looks very
similar to a function, but it is not quite the same thing.  Let me
show you an example:

@example
load supercrc;

type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    fun corrected_crc = int:
    @{
      try return calculate_crc (payload, control);
      catch if E_div_by_zero @{ return 0; @}
    @}

    int crc = corrected_crc;

    method c_crc = int:
    @{
      return corrected_crc;
    @}
  @};
@end example

We have added a method @code{c_crc} to our Packet struct type, that
just returns the corrected superCRC (patented, TM) of a packet.  This
can be invoked using dot-notation, once a Packet value is
mapped/constructed:

@example
(poke) var p = Packet @ 12#B
(poke) p.c_crc
0xdeadbeef
@end example

Now, the important bit here is that the method returns the corrected
crc @emph{of a @code{Packet}}.  That's it, it actually operates on a
Packet value.  This Packet value gets implicitly passed as an argument
whenever a method is invoked.

We can visualize this with the following ``pseudo Poke'':

@example
method c_crc = (Packet SELF) int:
@{
   return SELF.corrected_crc;
@}
@end example

Fortunately, poke takes care to recognize when you are referring to
fields of this implicit struct value, and does The Right Thing(TM) for
you.  This includes calling other methods:

@example
method foo = void: @{ ... @}
method bar = void:
@{
 [...]
 foo;
@}
@end example

The corresponding ``pseudo-poke'' being:

@example
method bar = (Packet SELF) void:
@{
 [...]
 SELF.foo ();
@}
@end example

It is also possible to define methods that modify the contents of
struct fields, no problem:

@example
var packet_special = 0xff;

type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size;
    [...]

    method set_size = (byte s) void:
    @{
      if (s == 0)
        size = packet_special;
      else
        size = s;
    @}
  @};
@end example

This is what is commonly known as a @dfn{setter}.  Note, incidentally,
how a method can also use regular variables.  The Poke compiler knows
when to generate a store in a normal variable such as
@code{packet_special}, and when to generate a set to a @code{SELF}
field.

Given the different nature of the variables, functions and methods,
there are a couple of restrictions:

@itemize @bullet
@item
Functions can't set fields defined in the struct type.

This will be rejected by the compiler:

@example
type Foo =
  struct
  @{
     int field;
     fun wrong = void: @{ field = 10; @}
  @};
@end example

Remember the construction/mapping process.  When a function accesses a
field of the struct type like in the example above, it is not doing
one of these pseudo @code{SELF.field = 10}.  Instead, it is simply
updating the value of the local created in this step in Foo_mapper:

@example
Foo_mapper:

 . Map an int, put it in a local `field'.
 . [...]
@end example

Setting that local would impact the mapping of the subsequent fields
if they refer to @code{field} (for example, in their constraint
expression) but it wouldn't actually alter the value of the field
@code{field} in the struct value that is created and returned from the
mapper!

This is very confusing, so we just disallow this with a compiler error
``invalid assignment to struct field'', for your own sanity 8-)

@item
Methods can't be used in field constraint expressions, nor in
variables or functions defined in a struct type.

How could they be?  The field constraint expressions, the
initialization expressions of variables, and the functions defined in
struct types are all executed as part of the mapper/constructor and,
at that time, there is no struct value yet to pass to the method.

If you try to do this, the compiler will greet you with an ``invalid
reference to struct method'' message.
@end itemize

Something to keep in mind about methods is that they can destroy the
integrity of the data stored in a struct.  Consider for example the
following struct type:

@example
type Packet =
  struct
  @{
    byte magic == 0xab;
    byte size : size <= 4096;
    [...]

    method set_size = (byte s) void:
    @{
      if (s == 0)
        size = packet_special;
      else
        size = s;
    @}
  @};
@end example

@noindent
Observe how this new version of @code{Packet} has an additional
constraint that specifies @code{size} should not exceed 4096.
However, when the method @code{set_size} is executed the constraints
are not checked again.  This is useful at times, but can also lead to
unintended data corruption.

A solution for this problem is to make methods aware of the
restrictions.  Like in this case:

@example
type Packet =
  struct
  @{
    var MAXSIZE = 4096;

    byte magic == 0xab;
    byte size : size <= MAXSIZE;
    [...]

    method set_size = (byte s) void:
    @{
      if (s == 0)
        size = packet_special;
      else if (s > MAXSIZE)
        raise E_inval;
      else
        size = s;
    @}
  @};
@end example

@noindent
Note how we use a variable @code{MAXSIZE} in order to avoid
hard-coding 4096 twice in the struct definition.

Occasionally it is useful to access to additional properties of a
containing struct or union in a method, other than its fields.  For
this purpose it is possible to refer explicitly to the struct using
the @code{SELF} keyword.  Note that the value referred has type
@code{any}.

For example, this is how we could print the name of the current
alternative in a pretty-printer for an union:

@example
type Foo =
  union int<32>
    @{
      int<32> One == 1;
      int<32> Two == 2;
      method _print = void:
      @{
        printf ("%s (%v)", SELF'ename (0), SELF'elem (0));
      @}
    @};
@end example

@node Padding and Alignment
@section Padding and Alignment
@cindex padding
@cindex alignment

It is often the case in binary formats that certain elements are
separated by some data that is not really used for any meaningful
purpose other than occupy that space.  The reason for keeping that
space varies from case to case; sometimes to reserve it for future
use, sometimes to make sure that the following data is aligned to some
particular alignment.  This is known as @dfn{padding}.  There are
several ways to implement padding in GNU poke.  This article shows
these techniques and discusses their advantages and disadvantages.

@menu
* Esoteric and exoteric padding::       External and internal padding.
* Reserved fields::                     Extra space not to be used.
* Payloads::                            Extra space to be used.
* Aligning struct fields::              Some stuff requires an alignment.
* Padding array elements::              An useful idiom.
@end menu

@node Esoteric and exoteric padding
@subsection Esoteric and exoteric padding

Padding is the technique of keeping some amount of space between two
different elements in some data stream.  GNU poke provides two
different ways to express sequences of data elements: the fields of a
struct type, which are defined one after the other, and elements in an
array.

We call adding space between two struct fields @dfn{esoteric} (or
internal) padding.

We call adding space between two array elements @dfn{exoteric} (or
external) padding.

The following sections contain examples of the two kinds of padding
and how to better handle them in Poke.

@node Reserved fields
@subsection Reserved fields

People designing binary encoded formats tend to be cautious and try to
avoid future backward incompatibilities by keeping some unused fields
that are reserved for future use.  This is the first kind of padding
we will be looking at, and is particularly common in structures like
headers.

See for example the header used to characterize compressed section
contents in ELF files:

@example
type Elf64_Chdr =
  struct
  @{
    Elf_Word ch_type;
    Elf_Word ch_reserved;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  @};
@end example

where the @code{ch_reserved} field is reserved for future use.  When
the time comes the space occupied by that field (32 bits in this case)
will be used to hold additional data in the form of one or more
fields.  The idea is that implementations of the older formats will
still work.

The most obvious way to handle this in Poke is using a named field
like @code{ch_reserved} above.  This field will be decoded/encoded by
poke when constructing/mapping/writing struct values of this type, and
will be available to the user as @code{chdr.ch_reserved}.

Sometimes reserved space is required to be filled with certain data
values, such as zeroes.  This may be to simplify things, or to force
data producers to initialize the memory in order to avoid potential
leaking of sensible information.  In these cases we can use Poke
initial values:

@example
type Elf64_Chdr =
  struct
  @{
    Elf_Word ch_type;
    Elf_Word ch_reserved == 0;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  @};
@end example

This will make poke to check that @code{ch_reserved} is zero when
constructing or mapping headers for compressed sections raising a
constraint violation exception otherwise.  It will also make poke to
make sure @code{ch_reserved} to 0 when constructing @code{Elf64_Chdr}
struct values:

@example
(poke) Elf64_Chdr @{ ch_reserved = 23 @}
unhandled constraint violation exception
@end example

An alternative way to characterize reserved space in Poke is to use
anonymous fields.  For example:

@example
type Elf64_Chdr =
  struct
  @{
    Elf_Word ch_type;
    Elf_Word;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  @};
@end example

Using Poke anonymous fields to implement reserved fields has at least
two advantages.  First, the user cannot anymore temper with the data
in the reserved space in an easy way, @i{i.e.} @code{chdr.ch_reserved
= 666} is no longer valid.  Second, the printed representation of
anonymous struct fields is more compact and denotes better than the
involved space is not to be messed with:

@example
(poke) Elf64_Chdr @{@}
Elf64_Chdr @{
  ch_type=0x0U,
  0x0U,
  ch_size=0x0UL#B,
  ch_addralign=0x0UL#B
@}
@end example

A disadvantage of using anonymous fields is that you cannot specify
constraint expressions for them, nor initial values.  At some point we
will probably add syntax to declare certain struct fields as
read-only.

At this point, it is important to note that anonymous fields are still
encoded/decoded by poke every time the struct value is mapped or
written, exactly like regular fields.  Therefore using them doesn't
pose any advantage in terms of performance.

@node Payloads
@subsection Payloads

The reserved fields discussed in the previous section are most often
discrete units like words, double-words, and the like, they are
usually of some fixed size, and they are used to delimit some space
that is not to be used.

Another kind of padding happens when an entity contains space to be
used to store some kind of payload whose contents are not determined.
This would be such an example:

@example
type Packet =
  struct
  @{
    offset<uint<32>,B> payload_size;
    byte[payload_size] payload;
    int flags;
  @};
@end example

In this example we are using a @code{payload} field which is an array
of bytes.  The size of the payload is determined by the packet header,
and the contents are not determined.  Of course this assumes that the
payload sizes are divisible in whole bytes; a bit-oriented format may
need to use an array of bits instead.

This approach of using a byte (or bit) array like in the example above
has the advantage of providing a field with the bytes (or bits) to the
user, for inspection and modification:

@example
(poke) packet.payload
[23UB, ...]
(poke) packet.payload[0] = 0
@end example

The user can still map whatever payload structure in that space using
the attributes of a mapped @code{Packet}.  For example, if the packet
contains an array of ULEB128 numbers, we could do:

@example
(poke) var numbers = ULEB128[packet.payload'size] @@ packet.payload'offset
@end example

But this approach has a disadvantage: every time the packet structure
is mapped or written the entire payload array gets decoded and
encoded.  If the payloads are big enough (think about the data blocks
of a file described by a file system i-node for example) this can be a
big problem in terms of performance.

Another problem of using byte (or bit) arrays for payloads is that the
printed representation of the struct values include the contents of
the arrays, and most often the user won't be interested in seeing
that:

@example
(poke) Packet @{ payload_size = 23#B @}
Packet @{
  payload_size=0x17U#B,
  payload=[0x0UB,0x0UB,0x0UB,0x0UB,0x0UB,...],
  flags=0x0
@}
@end example

Another alternative is to implement the padding implied by a payload
using field labels:

@example
type Packet =
  struct
  @{
    offset<uint<32>,B> payload_size;:
    int flags @@ OFFSET + payload_size;
  @};
@end example

  Note how a @code{payload} field no longer exists in the struct type,
  and the field @code{flags} is defined to start at offset
  @code{OFFSET + payload_size}.  This way no explicit array is
  encoded/decoded when manipulating @code{Packet} values:

@example
(poke) .set omaps yes
(poke) Packet @{ payload_size = 500#Mb @}
Packet @{
  payload_size=62500000U#B @@ 0UL#b,
  flags=0 @@ 4000000032UL#b
@} @@ 0UL#b
@end example

In this example we used the @code{omaps} option, which asks poke to
print the offsets of the fields.  The offset of @code{flags} is
4000000032 bits, or 500 megabytes:

@example
(poke) 4000000032UL #b/#MB
500UL
@end example

Mapping this new @code{Packet} involves reading and decoding five
bytes, for the @code{payload_size} and @code{flags} only.  This is
clearly much faster and avoids unneeded IO.

However you may be wondering, if there is no explicit @code{payload}
field, how to access the payload space?  A way is to define a method
to the struct to provide the payload attributes:

@example
type Packet =
  struct
  @{
    offset<uint<32>,B> payload_size;:
    var payload_offset = OFFSET;
    int flags @@ OFFSET + payload_size;

    method get_payload_offset = off64:
    @{
      return payload_offset;
    @}
  @};
@end example

Note how we captured the offset of the payload using a variable in the
strict type definition.  Returning @code{OFFSET} in
@code{get_payload_offset} wouldn't work for obvious reasons: in the
body of the method @code{OFFSET} evaluates to the end of @code{flags}
in this case.

Using this method you can easily access the payload (again as an array
of ULEB128 numbers) like this:

@example
var numbers = ULEB128[packet.payload_size @@ packet.get_payload_offset
@end example

Finally, using labels for this purpose makes the printed
representation of the struct values more readable by not including the
payload bytes in it:

@example
(poke) Packet @{@}
Packet @{
  payload_size=0x0U#B,
  flags=0x0
@}
@end example

@node Aligning struct fields
@subsection Aligning struct fields

Another kind of esoteric padding happens when certain fields in
entities are required to be aligned to some particular alignment.  For
example, suppose that the @code{flags} field in the packets used in
the previous sections is required to always be aligned to 4 bytes
regardless of the size of the payload.  This would be a common
requirement if the format is intended to be implemented in systems
where data is to be accessed using its ``natural'' alignment.

Using explicit fields for both the payload and the additional padding,
we could come with:

@example
type Packet =
  struct
  @{
    offset<uint<32>,B> payload_size;
    byte[payload_size] payload;
    byte[alignto (OFFSET, 4#B)] padding;
    int flags;
  @};
@end example

Where @code{alignto} is a little function defined in the Poke standard
library, like this:

@example
fun alignto = (uoff64 offset, uoff64 to) uoff64:
@{
  return (to - (offset % to)) % to;
@}
@end example

Alternatively, using the labels approach (which is generally better as
we discussed in the last section) the definition would become:

@example
type Packet =
  struct
  @{
    offset<uint<32>,B> payload_size;:
    var payload_offset = OFFSET;
    int flags @@ OFFSET + payload_size + alignto (payload_size, 4#B);

    method get_payload_offset = off64:
    @{
      return payload_offset;
    @}
  @};
@end example

In this case, the payload space is still completely characterized by
the @code{payload_size} field and the @code{get_payload_offset}
method.

@node Padding array elements
@subsection Padding array elements

Up to now all the examples of padding we have shown are in the
category of esoteric or internal padding, @i{i.e.} it was intended to
add space between fields of some particular entity.

However, sometimes we want to specify some padding between the
elements of a sequence of entities.  In Poke this basically means an
array.

Suppose we have a simple file system that is conformed by a sequence of
inodes.  The contents of the file system have the following form:

@example
+-----------------+
|      inode      |
+-----------------+
:                 :
:      data       :
:                 :
+-----------------+
|      inode      |
+-----------------+
:                 :
:      data       :
:                 :
+-----------------+
|      ...        |
@end example

That's it, each inode describes a block of data of variable size that
immediately follows.  Then more pairs of inode-data follow until the
end of the device.  However, a requirement is that each inode has to
be aligned to 128 bytes.

Let's start by writing a simple type definition for the inodes:

@example
type Inode =
  struct
  @{
    string filename;
    int perms;
    offset<uint<32>,B> data_size;
  @};
@end example

This definition is simple enough, but it doesn't allow us to just map
an array of inodes like this:

@example
(poke) Inode[] @ 0#B
@end example

We could of course add the data and padding explicitly to the inode
structure:

@example
type Inode =
  struct
  @{
    string filename;
    int perms;
    offset<uint<32>,B> data_size;
    byte[data_size] data;
    byte[alignto (data_size, 128#B) padding;
  @};
@end example

Then we could just map @code{Inode[] @@ 0#B} and we would the get expected
result.

But this is not a good idea.  On one hand because, as we know, this
would imply mapping the full file system data byte by byte, and that
would be very very slow.  On the other hand, because the data is not
part of the inode, conceptually speaking.

A better solution is to use this idiom:

@example
type Inode =
  struct
  @{
    string filename;
    int perms;
    offset<uint<32>,B> data_size;

    byte[0] @@ OFFSET + data_size + alignto (data_size, 128#B);
  @};
@end example

This uses an anonymous field at the end of the struct type, of size
zero, located at exactly the offset where the data plus padding would
end in the version with explicit fields.

This later solution is fast and still allows us to get an array of
inodes reflecting the whole file system with:

@example
(poke) var inodes = Inode[] @@ 0#B
@end example

Like in the previous sections, a method =get_data_offset= can be added
to the struct type in order to allow accessing the data blocks
corresponding to a given inode.

@node Dealing with Alternatives
@section Dealing with Alternatives

@subsection BSON

BSON @footnote{http://bsonspec.org/spec.html} is a binary encoding for
JSON documents.  The top-level entity described by the spec is the
@code{document}, which contains the total size of the document, a
sequence of elements, and a trailing null byte.

We can describe the above in Poke with the following type definition:

@example
type BSON_Doc =
  struct
  @{
    offset<int<32>,B> size;
    BSON_Elem[size - size'size - 1#B] elements;
    byte endmark : endmark == 0x0;
  @};
@end example

BSON elements come in different kinds, which correspond to the
different types of JSON entities: 32-bit integers, 64-bit integers,
strings, arrays, timestamps, and so on.  Every element starts with a
tag, which is a 8-bit unsigned integer that identifies it's kind, and
a name encoded as a NULL-terminated string.  What comes next depends
on the kind of element.

The following Poke type definition describes a subset of BSON
elements, namely integers, big integers and strings:

@example
type BSON_Elem =
  struct
  @{
    byte tag;
    string name;

    union
    @{
      int32 integer32 : tag == 0x10;
      int64 integer64 : tag == 0x12;
      BSON_String str : tag == 0x02;
    @} value;
  @};
@end example

The union in @code{BSON_Elem} corresponds to the variable part.  When
poke decodes an union, it tries to decode each alternative (union
field) in turn.  The first alternative that is successfully decoded
without raising a constraint violation exception is the chosen one.
If no alternative can be decoded, a constraint violation exception is
raised.

To see this process in action, let's use the BSON corresponding to the
following little JSON document:

@example
@{
    "name" : "Jose E. Marchesi",
    "age" : 40,
    "big" : 1076543210012345
@}
@end example

Let's take a look to the different elements:

@example
(poke) .load bson.pk
(poke) var d = BSON_Doc @ 0#B
(poke) d.elements'length
0x3UL
(poke) d.elements[0]
BSON_Elem @{
  tag=0x2UB,
  name="name",
  value=struct @{
    str=BSON_String @{
      size=0x11,
      value="Jose E. Marchesi",
      chars=[0x4aUB,0x6fUB,0x73UB,0x65UB...]
    @}
  @}
@}
(poke) d.elements[1]
BSON_Elem @{
  tag=0x10UB,
  name="age",
  value=struct @{
    integer32=0x27
  @}
@}
(poke) d.elements[2]
BSON_Elem @{
  tag=0x12UB,
  name="big",
  value=struct @{
    integer64=0x3d31c3f9e3eb9L
  @}
@}
@end example

Note how unions decode into structs featuring different fields. What
field is available depends on the alternative chosen while decoding.

In the session above, @code{d.elements[1].value} contains an
@code{integer32} field, whereas @code{d.elements[2].value} contains an
@code{integer64} field. Let's see what happens if we try to access the
wrong field:

@example
(poke) d.elements[1].value.integer64
unhandled invalid element exception
@end example

We get a run-time exception. This kind of errors cannot be caught at
compile time, since both @code{integer32} and @code{integer64} are
potentially valid fields in the union value.

@subsection Unions are Tagged

Unlike in C, Poke unions are tagged.  Unions have their own lexical
closures, and it is the capured values that determine what field is
chosen at every time.  Wherever the union goes, its tag accompanies
it.

To see this more clearly, consider the following alternative Poke
description of the BSON elements:

@example
type BSON_Elem =
  union
  @{
    struct
    @{
      byte tag : tag == 0x10;
      string name;
      int32 value;
    @} integer32;

    struct
    @{
      byte tag : tag == 0x12;
      string name;
      int64 value;
    @} integer64;

    struct
    @{
      byte tag : tag == 0x12;
      string name;
      BSON_String value;
    @} str;
  @};
@end example

This description is way more verbose than the one used in previous
sections, but it shows a few important properties of Poke unions.

First, the constraints guiding the decoding are not required to appear
in the union itself: it is a recursive process.  In this example,
@code{BSON_String} could have constraints on it's own, and these
constraints will also impact the decoding of the union.

Second, there are generally many different ways to express the same
plain binary using different type structures.  This is no different
than getting different parse trees from the same sequence of tokens
using different grammars denoting the same language.

See how different a BSON element looks (and feels) using this
alternative description:

@example
(poke) d.elements[0]
BSON_Elem @{
  str=struct @{
    tag=0x2UB,
    name="name",
    value=BSON_String @{
      size=0x11,
      value="Jose E. Marchesi",
      chars=[0x4aUB,0x6fUB,0x73UB,0x65UB...]
    @}
  @}
@}
(poke) d.elements[1]
BSON_Elem @{
  integer32=struct @{
    tag=0x10UB,
    name="age",
    value=0x27
  @}
@}
(poke) d.elements[2]
BSON_Elem @{
  integer64=struct @{
    tag=0x12UB,
    name="big",
    value=0x3d31c3f9e3eb9L
  @}
@}
@end example

What is the best way? It certainly depends on the kind of data you
want to manipulate, and the level of abstraction you want to achieve.
Ultimately, it is up to you.

Generally speaking, the best structuring is the one that allows you to
manipulate the data in terms of the structured abstractions as
naturally as possible.  That's the art and craft of writing good
pickles.

@node Structured Integers
@section Structured Integers


When we structure data using Poke structs, arrays and the like, we
often use the same structure than a C programmer would use.  For
example, to model ELF RELA structures, which are defined in C like:

@example
typedef struct
@{
  Elf64_Addr   r_offset;  /* Address */
  Elf64_Xword  r_info;    /* Relocation type and symbol index */
  Elf64_Sxword r_addend;  /* Addend */
@} Elf64_Rela;
@end example

@noindent
we could use something like this in Poke:

@example
type Elf64_Rela =
  struct
  @{
    Elf64_Addr r_offset;
    Elf64_Xword r_info;
    Elf64_Sxword r_addend;
  @};
@end example

Here the Poke struct type is pretty equivalent to the C incarnation.
In both cases the fields are always stored in the given order,
regardless of endianness or any other consideration.

However, there are situations where stored integral values are to be
interpreted as composite data.  This is the case of the @code{r_info}
field above, which is a 64-bit unsigned integer (@code{Elf64_Xword})
which is itself composed by several fields, depicted here:

@example
 63                                          0
+----------------------+----------------------+
|       r_sym          |      r_type          |
+----------------------+----------------------+
MSB                                         LSB
@end example

In order to support this kind of composition of integers, C
programmers usually resort to either bit masking (most often) or to
the often obscure and undefined behaviour-prone C bit fields.  In the
case of ELF, the GNU implementations define a few macros to access
these ``sub-fields'':

@example
#define ELF64_R_SYM(i)         ((i) >> 32)
#define ELF64_R_TYPE(i)        ((i) & 0xffffffff)
#define ELF64_R_INFO(sym,type) ((((Elf64_Xword) (sym)) << 32) + (type))
@end example

Where @code{ELF64_R_SYM} and @code{ELF64_R_TYPE} are used to extract
the fields from an @code{r_info}, and @code{ELF64_R_INFO} is used to
compose it.  This is typical of C data structures.

We could of course mimic the C implementation in Poke:

@example
fun Elf64_R_Sym = (Elf64_Xword i) uint<32>:
   @{ return i .>> 32; @}
fun Elf64_R_Type = (Elf64_Xword i) uint<32>:
   @{ return i & 0xffff_ffff; @}
fun Elf64_R_Info = (uint<32> sym, uint<32> type) Elf64_Xword:
   @{ return sym as Elf64_Xword <<. 32 + type; @}
@end example

However, this approach has a huge disadvantage: since we are not able
to encode the logic of these ``sub-fields'' in proper Poke fields,
they become second class citizens, with all that implies: no
constraints on their own, can't be auto-completed, can't be assigned
individually, @i{etc}.

But we can use the so-called @dfn{integral structs}!  These are
structs that are defined exactly like your garden variety Poke
structs, with a small addition:

@example
type Elf64_RelInfo =
  struct uint<64>
  @{
    uint<32> r_sym;
    uint<32> r_type;
  @};
@end example

Note the @code{uint<64>} addition after @code{struct}.  This can be
any integer type (signed or unsigned).  The fields of an integral
struct should be integral themselves (this includes both integers and
offsets) and the total size occupied by the fields should be the same
size than the one declared in the struct's integer type.  This is
checked and enforced by the compiler.

The Elf64 RELA in Poke can then be encoded like:

@example
type Elf64_Rela =
  struct
  @{
    Elf64_Addr r_offset;
    struct Elf64_Xword
    @{
      uint<32> r_sym;
      uint<32> r_type;
    @} r_info;
    Elf64_Sxword r_addend;
  @};
@end example

When an integral struct is mapped from some IO space, the total number
of bytes occupied by the struct is read as a single integer value, and
then the values of the fields are extracted from it.  A similar
process is using when writing.  That is what makes it different with
respect a normal Poke struct.

It is possible to obtain the integral value corresponding to an
integral struct using a cast to an integral type:

@example
(poke) type Foo = struct int<32> @{ int<16> hi; uint<16> lo; @};
(poke) Foo @{ hi = 1 @} as int<32>
0x10000
@end example

An useful idiom, that doesn't require to specify an explicit integral
type, is this:

@example
(poke) type Foo = struct int<32> @{ int<16> hi; uint<16> lo; @};
(poke) var x = Foo @@ 0#B;
(poke) +x
0xfe00aa
@end example

These casts allow to ``integrate'' struct values explicitly, but the
compiler also implicitly promotes integral struct values to integers
in all contexts where an integer is expected:

@itemize
@item Operator to an addition, subtraction, multiplication, division,
ceil division, bit left shift, bit right shift, bit-wise not, bit-wise
and, bit-wise or, bit-wise xor, logical and, logical or, logical not,
logical implication, positive, negative, bit-concatenation, condition
in conditional expression.
@item Argument to a function whose formal expects an integer.
@item Value returned from a function that returns an integer.
@item Condition in loop statement.
@item Constraint expression in a struct field.
@item Expression in a conditional struct field.
@end itemize

Note that the above list doesn't include relational operators, since
these operators work with struct values.

@node Working with Incorrect Data
@section Working with Incorrect Data

We have seen how Poke type definitions very often include constraint
expressions reflecting the integrity of the data stored in these
types.

Consider for example the following abbreviated description of the
header of an ELF file:

@example
type Elf64_Ehdr =
  struct
  @{
    struct
    @{
      byte[4] ei_mag = [0x7fUB, 'E', 'L', 'F'];
      byte ei_class;
      [...]
    @} e_ident;

    Elf_Half e_type;
    Elf_Half e_machine;
    Elf_Word e_version = EV_CURRENT;
    [...]
    Elf_Half e_shstrndx : e_shnum == 0 || e_shstrndx < e_shnum;
  @};
@end example

@noindent
There are three constraint expressions in the definition above:

@itemize
@item The constraint expression in the field @code{ei_mag} makes sure that a
right magic number begins the header.
@item The constraint expression in @code{e_version} checks that the ELF version is the current version.
@item The constraint expression in @code{e_shstrndx} checks that
the index stored in the field doesn't overflow the section header
table of the file.
@end itemize

Every time we construct or map an @code{Elf64_Ehdr} value the
constraint expressions are checked.  In case any of the constraints
are not satisfied we get a ``constraint violation exception''.  This
is useful to realize that we are facing invalid data (when mapping) or
that we are trying to build an invalid ELF header (when constructing.)

However, the exception avoids the completion of the mapping or
construction operation.  Sometimes this is inconvenient:

@itemize
@item When we are given invalid data that we want to explore or fix.
Imagine for example a programmer investigating a possible assembler or
linker bug that results in an invalid ELF file.
@item When we are discovering or inferring the structure of some data:
we suspect at some point there may be something that is similar to an
ELF header.
@item When we want to create invalid data.  Suppose for example a
programmer who is building a test case for the linker, that requires a
corrupted ELF file.
@end itemize

The way poke provides support for these situation is using the concept
of @dfn{map strictness}.  Suppose we map an ELF header in some IO
space:

@example
(poke) load elf
(poke) var ehdr = Elf64_Ehdr @@ 0#B
@end example

@noindent
The mapping operator @code{@@} performs a @dfn{strict mapping}.  This
means that constraint expressions are evaluated and the operation is
interrupted if any constraint fails, as described above.  Also, this
means the integrity will be checked when the data is modified:

@example
(poke) ehdr.e_version = 666
unhandled constraint violation exception
@end example

@noindent
So the mapping operator generates ``strict'' values.  We can check the
strictness of a mapped value using the @code{'strict} attribute:

@example
(poke) ehdr'strict
1
@end example

If we know that we are facing corrupted data, or if we want to corrupt
the ELF header, we perform a @dfn{non-strict mapping} using a variant
of the mapping operator:

@example
(poke) var ehdr_ns = Elf64_Ehdr @@! 0#B
(poke) ehdr_ns'strict
0
@end example

@noindent
When mapping is performend in non-strict mode, the constraint
expressions are still executed, but their resulting value is
discarded.  The constraint expressions are executed regardless of
strictness because it is fairly common for them to include
side-effects that have a calculated impact in subsequent operations,
such as for example setting the endianness or some other global state.

We can now corrupt the header in the example above:

@example
(poke) ehdr_ns.e_version = 666
@end example

@node Maps and Map-files
@chapter Maps and Map-files

Editing data with GNU poke mainly involves creating mapped values and
storing them in Poke variables.  However, this may not be that
convenient when poking several files simultaneously, and when the
complexity of the data increases.  poke provides a convenient
mechanism for this: maps and map files.

@menu
* Editing using Variables::     Data is usually edited mapping variables.
* poke Maps::                   Maps and map-files.
* Loading Maps::                Using maps defined in files.
* Multiple Maps::               Multiple perspectives of the same data.
* Auto-map::                    Loading maps automagically.
* Constructing Maps::           Creating and managing maps on the fly.
* Predefined Maps::             Collection of already-written maps.
@end menu

@node Editing using Variables
@section Editing using Variables

Editing data with GNU poke mainly involves creating mapped values and
storing them in Poke variables.  For example, if we were interested in
altering the fields of the header in an ELF file, we would map an
@code{Elf64_Ehdr} struct at the beginning of the underlying IO space
(the file), like in:

@example
(poke) .file foo.o
(poke) load elf
(poke) var ehdr = Elf64_Ehdr @@ 0#B
@end example

At this point the variable @code{ehdr} holds an @code{Elf64_Ehdr}
structure, which is mapped.  As such, altering any of the fields of
the struct will update the corresponding bytes in @code{foo.o}.  For
example:

@example
(poke) ehdr.e_entry = 0#B
@end example

A Poke value has three mapping related attributes: whether it is
mapped, the offset at which it is mapped in an IO space, and in which
IO space.  This information is accessible for both the user and Poke
programs using the following attributes:

@example
(poke) ehdr'mapped
1
(poke) ehdr'offset
0UL#b
(poke) ehdr'ios
0
@end example

That's it, @code{ehdr} is mapped at offset zero byte in the IO space
@code{#0}, which corresponds to @code{foo.o}:

@example
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #0	FILE	rw	0x00000000#B	0x000004c8#B   ./foo.o
@end example

Now that we have the ELF header, we may use it to get access to the
ELF section header table in the file, that we will reference using
another variable @code{shdr}:

@example
(poke) var shdr = Elf64_Shdr[ehdr.e_shnum] @@ ehdr.e_shoff
(poke) shdr[1]
Elf64_Shdr @{
  sh_name=0x1bU#B,
  sh_type=0x1U,
  sh_flags=#<ALLOC,EXECINSTR>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0xbUL#B,
  sh_link=0x0U,
  sh_info=0x0U,
  sh_addralign=0x1UL,
  sh_entsize=0x0UL#b
@}
@end example

Variables are convenient entities to manipulate in Poke.  Let's
suppose that the file has a lot of sections and we want to do some
transformation in every section.  It is a time consuming operation,
and we may forget which sections we have already processed and which
not. We could create an empty array to hold the sections already
processed:

@example
(poke) var processed = Elf64_Shdr[] ()
@end example

And then, once we have processed some given section, add it to the
array:

@example
... edit shdr[23] ...
(poke) processed += [shdr[23]]
@end example

Note how the array =processed= is not mapped, but the sections
contained in it are mapped: Poke uses copy by shared value.  So, after
we spend the day carefully poking our ELF file, we can ask poke, are
we done with all the sections in the file?

@example
(poke) shdr'length == processed'length
1
@end example

Yes, we are.  This can be made as sophisticated as desired.  We could
easily write a function that saves the contents of @code{processed} in
files, so we can continue hacking tomorrow, for example.

We can then concluding that using mapped variables to edit data
structures stored in IO spaces works well in common and simple cases
like the above: we make our ways mapping here and there, defining
variables to hold data that interests us, and it is easy to remember
that the variables @code{ehdr} and @code{shdr} are mapped, where are
they mapped, and that they are mapped in the file @code{foo.o}.

However, GNU poke allows to edit more than one IO space
simultaneously.  Let's say we now want to poke the sections of another
ELF file: @code{bar.o}.  We would start by opening the file:

@example
(poke) .file bar.o
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	FILE	rw	0x00000000#B	0x000004c8#B	./bar.o
  #0	FILE	rw	0x00000000#B	0x000004c8#B	./foo.o
@end example

Now that @code{bar.o} is the current IO space, we can map its header.
But now, what variable to use?  We would rather not redefine
@code{ehdr}, because that is already holding the header of
@code{foo.o}.  We could adapt our naming schema on the fly:

@example
(poke) var foo_ehdr = ehdr
(poke) var bar_ehdr = Elf64_Ehdr @@ 0#B
@end example

But then we would need to do the same for the other variables too:

@example
(poke) var foo_shdr = shdr
(poke) var bar_shdr = Elf64_Shdr[bar_ehdr.e_shnum] @@ bar_ehdr.e_shoff
@end example

However, we can easily see how this can degenerate quickly: what about
@code{processed}, for example?  In general, as the number of IO spaces
being edited increases it becomes more and more difficult to manage
our mapped variables, which are associated to each IO space.

@node poke Maps
@section poke Maps

As we have seen mapping variables is a very powerful, general and
flexible mean to edit stored binary data in one or more IO spaces.
However it is easy to lose track of where the variables are mapped
and, ideally speaking, we would want to have a mean to refer to, say,
the ``ELF header'', and get the header as a mapped value regardless of
what specific file we are editing.  Sort of a ``meta variable''.  GNU
poke provides a way to do this: @dfn{maps}.

A @dfn{map} can be conceived as a sort of ``view'' that can be applied
to a given IO space.  Maps have entries, which are values mapped at
some given offset, under certain conditions.  For example, we have
seen an ELF file contains, among other things, a header at the
beginning of the file and a table of section headers of certain size
and located at certain location determined by the header.  These would
be two entries of a so-called ELF map.

poke maps are defined in @dfn{map files}.  These files use the
@code{.map} extension.  A map file @code{self.map} (for
sectioned/simple elf) defining the view of an ELF file as a header and
a table of section header would look like this:

@example
/* self.map - map file for a simplified view of an ELF file.  */

load elf;

%%

%entry
%name ehdr
%type Elf64_Ehdr
%offset 0#B

%entry
%name shdr
%type Elf64_Shdr[(Elf64_Ehdr @@ 0#B).e_shnum]
%condition (Elf64_Ehdr @@ 0#B).e_shnum > 0
%offset (Elf64_Ehdr @@ 0#B).e_shoff
@end example

This map file defines a view of an ELF file as a header entry
@code{ehdr} and an entry with a table of section headers @code{shdr}.

The first section of the file, which spans until the separator line
containing @code{%%}, is arbitrary Poke code which as we shall see,
gets evaluated before the map entries are processed.  This is called
the map "prologue".  In this case, the prologue contains a comment
explaining the purpose of the file, and a single statement @code{load}
that loads the @code{elf.pk} pickle, since the entries below use
definitions like @code{Elf64_Ehdr} that are defined by that pickle.
The prologue is useful to define Poke functions and other entities
that are then used in the definitions of the entries.

A separator line containing only @code{%%} separates the prologue from
the next section, which is a list of entries definitions.  Each entry
definition starts with a line @code{%entry}, and has the following
attributes:

@table @code
@item %name
Like @code{ehdr} and @code{shdr}.  These names should follow the same
rules than Poke variables, but as we shall see later, map entries are
not Poke variables.  This attribute is mandatory.

@item %type
This can be any Poke expression denoting a type, like @code{int},
@code{Elf64_Ehdr} or @code{Elf64_Shdr[(Elf64_Ehdr @@ 0#B).e_shnum]}.
This attribute is mandatory.

@item %condition
If specified, will determine whether to include the entry in the map.
In the example above, the map will have an entry @code{shdr} only if the
ELF file has one or more sections.  Any Poke expression evaluating to
a boolean can be used as conditions.  This attribute is optional:
entries not having a condition will always be included in the map.

@item %offset

Offset in the IO space where the entry will be mapped.  Any Poke
expression evaluating to an offset can be used as entry offset.  This
attribute is mandatory.
@end table

@node Loading Maps
@section Loading Maps

So we have written our =self.map=, which denotes a view or structure
of ELF files we are interested on, and that resides in the current
working directory.  How to use it?

The first step is to fire up poke and open some object file.  Let's
start with @code{foo.o}:

@example
(poke) .file foo.o
@end example

@noindent
Now, we can load the map using the @code{.map load} dot-command:

@example
(poke) .map load self
[self](poke)
@end example

The @code{.map load self} command makes poke to look in certain
directories for a file called @code{self.map}, and to load it.  The
list of directories where poke looks for map files is encoded in the
variable @code{map_load_path} as a string containing a maybe empty
list of directories separated by @code{:} characters.  Each directory
is tried in turn.  This variable is initialized with suitable
defaults:

@example
(poke) map_load_path
"/home/jemarch/.poke.d:.:/home/jemarch/.local/share/poke:[...]"
@end example

Once a map is loaded, observe how the prompt changed to contain a
prefix @code{[self]}.  This means that the map @code{self} is loaded
for the current IO space.  You can choose to not see this information
in the prompt by setting the @code{prompt-maps} option either at the
prompt or in your @file{.pokerc}:

@example
poke) .set prompt-maps no
@end example

By default @code{prompt-maps} is @code{yes}.  This prompt aid is
intended to provide a cursory look of the "views" or maps loaded for
the current IO space.  If we load another IO space and switch to it,
the prompt changes accordingly:

@example
(poke) [self](poke) .mem foo
The current IOS is now `*foo*'.
(poke) .ios #0
The current IOS is now `./foo.o'.
[self](poke)
@end example

At any time the @code{.info maps} dot-command can be used to obtain a
full list of loaded maps, with more information about them:

@example
(poke) .info maps
IOS   Name   Source
#0    self   ./self.map
@end example

In this case, there is a map @code{self} loaded in the IO space
@code{#0}, which corresponds to @code{foo.o}.

Once we make @code{foo.o} our current IO space, we can ask poke to
show us the entries corresponding to this map using another
dot-command:

@example
    : (poke) .map show self
    : Offset     Entry
    : 0x0UL#B    $self::ehdr
    : 0x208UL#B  $self::shdr
@end example

This tells us there are two entries for @code{self} in @code{foo.o}:
@code{$self::ehdr} and @code{$self::shdr=} Note how map entries use
names that start with the @code{$} character, then contain the name of
the map an the name of the entry we defined in the map file, separated
by @code{::}.

We can now use these entries at the prompt like if they were regular
mapped variables:

@example
[self](poke) $self::ehdr
Elf64_Ehdr @{
  e_ident=struct @{
    ei_mag=[0x7fUB,0x45UB,0x4cUB,0x46UB],
    [...]
  @},
 e_type=0x1UH,
 e_machine=0x3eUH,
 [...]
@}
(poke) $self::shdr'length
11UL
@end example

It is important to note, however, that map entries like
@code{$foo::bar} are @emph{not} part of the Poke language, and are
only available when using poke interactively.  Poke programs and
scripts can't use them.

Let's now open another ELF file, and the =self= map in it:

@example
(poke) .file /usr/local/lib/libpoke.so.0.0.0
(poke) .map load self
[self](poke)
@end example

So now we have two ELF files loaded in poke: @code{foo.o} and
@code{libpoke.so.0.0.0}, and in both IO spaces we have the @code{self}
map loaded.  We can easily see that the map entries are different
depending on the current IO space:

@example
[self](poke) .map show self
Offset       Entry
0UL#B        $self::ehdr
3158952UL#B  $self::shdr
[self](poke) .ios #0
The current IOS is now `./foo.o'.
[self](poke) .map show self
Offset   Entry
0UL#B    $self::ehdr
520UL#B  $self::shdr
@end example

@noindent
@code{foo.o} is an object file, whereas @code{libpoke.so.0.0.0} is a
DSO:

@example
(poke) .ios #0
The current IOS is now `./foo.o'.
[self](poke) $self::ehdr.e_type
1UH
[self](poke) .ios #2
The current IOS is now `/usr/local/lib/libpoke.so.0.0.0'.
[self](poke) $self::ehdr.e_type
3UH
@end example

The interpretation of the map entry @code{$self::ehdr} is different
depending on the current IO space.  This makes it possible to refer to
the ``ELF header'' of the current file.

Underneath, poke implements this by defining mapped variables and
``redirecting'' the entry names @code{$foo::bar} to the right variable
depending on the IO space that is currently selected.  It hides all
that complexity from us.

@node Multiple Maps
@section Multiple Maps

It is perfectly possible (and useful!) to load more than one map in
the same IO space.  It is very natural for a single file, for example,
to contain data that can be interpreted in several ways, or of
different nature.

Let's for example open again an ELF file, this time compiled with
@code{-g}:

@example
(poke) .file foo.o
@end example

We now load our @code{self} map, to get a view of the file as a collection
of sections:

@example
(poke) .map load self
[self](poke)
@end example

And now we load the @code{dwarf} map that comes with poke, to get a
view of the file as having debugging information encoded in DWARF:

@example
[self(poke) .map load dwarf
[dwarf,self](poke)
@end example

See how the prompt now reflects the fact that the current IO space
contains DWARF info!  Let's take a look:

@example
[dwarf,self](poke) .info maps
IOS   Name    Source
#0    dwarf   /home/jemarch/gnu/hacks/poke/maps/dwarf.map
#0    self    ./self.map
[dwarf,self](poke) .map show dwarf
Offset    Entry
0x5bUL#B  $dwarf::info
@end example

Now we can access entries from any of the loaded maps, @i{i.e.} access
the file in terms of different perspectives.  As an ELF file:

@example
[dwarf,self](poke) $self::shdr[1]
Elf64_Shdr @{
  sh_name=0xb5U#B,
  sh_type=0x11U,
  sh_flags=#<>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0x8UL#B,
  sh_link=0x18U,
  sh_info=0xfU,
  sh_addralign=0x4UL,
  sh_entsize=0x4UL#b
@}
@end example

@noindent
And as a file containing DWARF info:

@example
[dwarf,self](poke) $dwarf::info
Dwarf_CU_Header @{
  unit_length=#<0x0000004eU#B>,
  version=0x4UH,
  debug_abbrev_offset=#<0x00000000U#B>,
  address_size=0x8UB#B
@}
@end example

If you are curious about how the DWARF entries are defined, look at
@code{maps/dwarf.map} in the poke source distribution, or in your
installed poke (@code{.info maps} will tell you the file the map got
loaded from.)

It is possible to unload or remove a map from a given IO space using
the @code{.map remove} dot-command.  Say we are done looking at the
DWARF in @file{foo.o}, and we are no longer interested in it as a file
containing debugging info.  We can do:

@example
[dwarf,self](poke) .map remove dwarf
[self](poke)
@end example

Note how the prompt was updated accordingly: only @code{self} remains
as a loaded map on this file.

@node Auto-map
@section Auto-map

Certain maps make sense when editing certain types of data.  For
example, @file{dwarf.map} is intended to be used in ELF files.  In
order to ease using maps, poke provides a feature called @dfn{auto
mapping}, which is disabled by default.

@noindent
You can set auto mapping like this:

@example
(poke) .set auto-map yes
@end example

When auto mapping is enabled, poke will look to the value of the
pre-defined variable @code{auto_map}, which must contain an array of
pairs of strings, associating a regular expression with a map name.

For example, you may want to initialize @code{auto_map} like this in
your @file{.pokerc} file:

@example
auto_map = [[".*\\.mp3$", "mp3"],
            [".*\\.o$", "elf"],
            ["a\\.out$", "elf"]];
@end example

This will make poke to load =mp3.map= for every file whose name ends
with @file{.mp3}, and @file{elf.map} for files having names like
@file{foo.o} and @file{a.out}.

Following the usual pokeish philosophy of being as less as intrusive
by default as possible, the default value of @code{auto_map} is the
empty string.

@node Constructing Maps
@section Constructing Maps

As we have seen, we can define our own maps using map files like
@file{self.map}, which contain a prologue and a set of map entries.
However, sometimes it is useful to create maps ``on the fly'' while we
explore some data with poke.

To make this possible, poke provides a suitable set of dot-commands.
Let's say we are poking some data, and we want to create a map for it.
We can do that like this:

@example
(poke) .map create mymap
@end example

@noindent
This creates an empty map named @code{mymap}, with no entries:

@example
[mymap](poke) .map show mymap
Offset   Entry
@end example

Adding entries is easy.  First, we have to map some variable, and then
use it as the base for the new entry:

@example
[mymap](poke) var foo = int[3] @ 0#B
[mymap](poke) .map entry add mymap, foo
[mymap](poke) .map show mymap
Offset   Entry
0x0UL#B  $mymap::foo
@end example

Note how the entry @code{$mymap::foo} gets created, associated to the
current IO space and mapped at the same offset than the variable
@code{foo}.

We can remove entries from existing maps using the @code{.map entry
remove} dot-command:

@example
[mymap](poke) .map entry remove mymap, foo
[mymap](poke) .map show mymap
Offset   Entry
[mymap](poke)
@end example

We plan to add an additional command to save maps to map files.  The
idea is that you can create your maps on the fly, save them, and then
load them back some other day when you are ready to continue poking.
This is not implemented yet though.

@node Predefined Maps
@section Predefined Maps

GNU poke comes with a set of useful pre-written maps, which get
installed in a system location.  We want to expand this collection, so
please send us your map files!

@node Writing Pickles
@chapter Writing Pickles

GNU poke encourages the user to write little pieces of code in order
to face spontaneous needs and fix situations.  Is that name in the
file encoded in a fixed array of characters padded with white spaces?
No problem, just write a three lines function so you can update the
file using a comfortable NULL-terminated string.  Better than waiting
for some poke maintainer to add that function for you, isn't it?

Just save your functions in some personal @file{.pk} file that you
load at startup, and your magic tricks bag will increase in time,
making your poking more and more efficient.

However, when it comes to share the code with other people, it is
important to follow certain conventions in order to achieve certain
uniformity.  This makes it easier for other people to discover what
your hack provides, and how it works.  It is this consistency, and
these conventions, that makes some random @file{.pk} file a
@dfn{pickle}.

This guide contains guidelines and recommendations for the pickle's
writer.

@menu
* Pretty-printers::		Conventions for pretty-printed output.
* Setters and Getters::		Anatomy getters and setters.
@end menu

@node Pretty-printers
@section Pretty-printers

@subsection Convention for pretty-printed Output

Very often the structure of the data encoded in binary is not very
intelligible.  This is because it is usual for binary formats to be
designed with goals in mind other than being readable by humans:
compactness, detail @i{etc}.

In our pickle we of course want to provide access to the very finer
detail of the data structures.  However, we also want for the user to
be able to peruse the data visually, and only look at the fine detail
on demand.

Consider for example an ID3V1 tag data from some MP3 file.   This is
the result of mapping a @code{ID3V1_Tag}:

@example
ID3V1_Tag @{
  id=[0x54UB,0x41UB,0x47UB],
  title=[0x30UB,0x31UB,0x20UB,0x2dUB,0x20UB,...],
  artist=[0x4aUB,0x6fUB,0x61UB,0x71UB,0x75UB,...],
  album=[0x4dUB,0x65UB,0x6eUB,0x74UB,0x69UB,...],
  year=[0x20UB,0x20UB,0x20UB,0x20UB],
  data=struct @{
    extended=struct @{
      comment=[0x20UB,0x20UB,0x20UB,0x20UB,0x20UB,...],
      zero=0x0UB,
      track=0x1UB
    @}
  @},
  genre=0xffUB
@}
@end example

Not very revealing.  Fortunately, poke supports pretty printers.  If a
struct type has a method called @code{_print}, it will be used by poke
as a pretty printer if the @option{pretty-print} option is set:

@example
(poke) .set pretty-print yes
@end example

By convention, the output of pretty-printers should always start with
@code{#<} and end with @code{>}.  The convention makes it explicit for
the user that everything she sees between @code{#<} and @code{>} is
pretty-printed, and do @emph{not} necessarily reflect the physical
structure of the data.  Also some information may be missing.  In
order to get an exact and complete description of the data, the user
should @command{.set pretty-print no} and evaluate the value again at
the prompt.

For example, in the following BPF instructions it is obvious at first
sight that the shown register values are pretty-printed:

@example
BPF_Insn = @{
  ...
  regs=BPF_Regs @{
     src=#<%r3>,
     dst=#<%r0>
  @}
  ...
@}
@end example

If the pretty-printed representation spans for more than one line,
please place the opening @code{#<} in its own line, then the lines
with the data, and finally @code{>} in its own line, starting at
column 0.

Example of the MP3 tag above, this time pretty-printed:

@example
#<
  genre: 255
  title: 01 - Eclipse De Mar
  artist: Joaquin Sabina
  album: Mentiras Piadosas
  year:
  comment:
  track: 1
>
@end example

@subsection Pretty Printing Optional Fields

Let's say we are writing a pretty-printer method for a struct type
that has an optional field.  Like for example:

@example
type Packet =
  struct
  @{
    byte magic : magic in [MAGIC1,MAGIC2];
    byte n;
    byte[n] payload;
    if (magic == MAGIC2)
     PacketTrailer trailer;
  @}
@end example

@noindent
In this case, the struct value will have a @code{trailer}
conditionally, which has to be tackled on the pretty-printer
somehow.

An approach that often works good is to replicate the logic in the
optional field condition expression, like this:

@example
  struct
  @{
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    PacketTrailer trailer if packet_magic2_p (magic);

    method _print = void:
    @{
      [...]
      if (magic == MAGIC2)
        pretty_print_trailer;
    @}
  @}
@end example

@noindent
This works well in this simple example.  In case the expression is big
and complicated, we can avoid rewriting the same expression by
encapsulating the logic in a function:

@example
fun packet_magic2_p = int: @{ return magic == MAGIC2; @}
type Packet =
  struct
  @{
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    if (packet_magic2_p (magic))
    PacketTrailer trailer;

    method _print = void:
    @{
      [...]
      if (packet_magic2_p (magic))
        pretty_print_trailer;
    @}
  @}
@end example

@noindent
However, this may feel weird, as the internal logic of the type
somehow leaks its natural boundary into the external function
@code{packet_magic2_p}.

An alternative is to use the following idiom, that checks whether the
field actually exists in the struct:

@example
type Packet =
  struct
  @{
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    if (packet_magic2_p (magic))
      PacketTrailer trailer;

    method _print = void:
    @{
      [...]
      try pretty_print_trailer;
      catch if E_elem @{@}
    @}
  @}
@end example

@noindent
This approach also works with unions:

@example
type ID3V1_Tag =
  struct
  @{
    [...]
    union
    @{
      /* ID3v1.1  */
      struct
      @{
        char[28] comment;
        byte zero = 0;
        byte track : track != 0;
      @} extended;
      /* ID3v1  */
      char[30] comment;
    @} data;
    [...]

    method _print = void:
    @{
      [...]
      try print "  comment: " + catos (data.comment) + "\n";
      catch if E_elem
      @{
        print "  comment: " + catos (data.extended.comment) + "\n";
        printf "  track: %u8d", data.extended.track;
      @}
    @}
  @}
@end example

@node Setters and Getters
@section Setters and Getters

Given a struct value, the obvious way to access the value of a field
is to just refer to it using dot-notation.

For example, for the following struct type:

@example
type ID3V1_Tag =
  struct
  @{
    [...]
    char[30] title;
    char[30] artist;
    char[30] album;
    char[4] year;
    [...]
  @}
@end example

Suppose the find out the year in the tag is wrong, off by two years:
the song was release in 1980, not in 1978!.  Unfortunately, due to the
bizarre way the year is stored in the file (as a sequence of digits
encoded in ASCII, non-NULL terminated) we cannot just write:

@example
(poke) tag.year = tag.year + 2
error
@end example

Instead, we can use facilities from the standard library and a bit of
programming:

@example
(poke) stoca (format ("%d", atoi (catos (tag.year)) + 2), tag.year)
@end example

The above line basically transforms the data we want to operate on
(the tag year) from the stored representation into a more useful
representation (from an array of character digits to an integer) then
operates with it (adds two) then converts back to the stored
representation.  Let's call this ``more useful'' representation the
``preferred representation''.

A well written pickle should provide @dfn{getter} and @dfn{setter}
methods for fields in struct types for which the stored representation
is not the preferred representation.  By convention, getter and setter
methods have the following form:

@example
method get_@var{field} @var{preferred_type}: @{ ... @}
method set_@var{field} (@var{preferred_type} val) void: @{ ... @}
@end example

@noindent
Using the @code{get_} and @code{set_} prefixes consistently is very
important, because the pokist using your pickle can easily find out
the available methods for some given value using tab-completion in the
REPL.

For example, let's add setter and getter methods for the field
@code{year} in the ID3V1 tag struct above:

@example
type ID3V1_Tag =
  struct
  @{
    [...]
    char[4] year;

    method get_year = int: @{ return atoi (catos (year)); @}
    method set_year = (int val) void:
    @{
      var str = format ("%d", val);
      stoca (str, year);
    @}
    [...]
  @}
@end example

What constitutes the preferred representation of a field is up to the
criteria of the pickle writer.  For the tag above, I would say the
preferred representations for the title, artist, album and year are
string for title, artist and album, and an integer for the year.

@node Writing Binary Utilities
@chapter Writing Binary Utilities

GNU poke is, first and foremost, intended to be used as an interactive
editor, either directly on the command line or using a graphical user
interface built on it.  However, since its conception poke was
intended to also provide a suitable and useful foundation on which
other programs, the so-called @dfn{binary utilities}, could be
written.  This chapter shows how to write Poke scripts and programs.

@menu
* Poke Scripts::                Using poke as an interpreter.
* Command-Line Arguments::      Handling command-line arguments.
* Exiting from Scripts::        Exit with an error status.
* Loading pickles as Modules::  Importing pickles in programs.
* elfextractor::                An example Poke binary utility.
* Filters::                     Writing filters for binary data.
@end menu

@node Poke Scripts
@section Poke Scripts

In interactive usage, there are two main ways to execute Poke code: at
the interactive prompt (or REPL), and loading ``pickles''.

Executing Poke code at the REPL is as easy as introducing a statement
or expression:

@example
(poke) print "Hello\n"
Hello
@end example

Executing Poke code in a pickle is performed by loading the file
containing the code:

@example
(poke) .load say-hello.pk
Hello
@end example

@noindent
Where @file{say-hello.pk} contains simply:

@example
print "Hello\n";
@end example

However, we would like to have Poke scripts, @i{i.e}. to be able to
execute Poke programs as standalone programs, from the shell.  In
other words, we want to use GNU poke as an interpreter.  This is
achieved by using a shebang, which should appear at the top of the
script file.  The poke shebang looks like this:

@example
#!/usr/bin/poke -L
!#
@end example

The @option{-L} command line option tells poke that it is being used
as an interpreter.  Additional arguments for poke can be specified
before @option{-L} (but not after).  The @code{#!  ... !#} is an
alternative syntax for multi-line comments, which allows to have the
shebang at the top of a Poke program without causing a syntax error.
This nice trick has been borrowed from guile.

Therefore, we could write say-hello as a Poke script like this:

@example
#!/usr/bin/poke -L
!#

print "Hello\n";
@end example

@noindent
And then execute it like any other program or script:

@example
$ ./say-hello
@end example

@node Command-Line Arguments
@section Command-Line Arguments

When a Poke script is executed, the command line arguments passed to
the script become available in the array argv.  Example:

@example
#!/usr/bin/poke -L
!#

for (arg in argv)
  print "Argument: " + arg + "\n";
@end example

@noindent
Executing this script results in:

@example
$ ./printargs foo bar 'baz quux'
Argument: foo
Argument: bar
Argument: baz quux
@end example

Note how it is not needed to have an argc variable, since the number
of elements stored in a Poke array can be queried using an attribute:
@code{argv'length}.

Note also that @code{argv} is only defined when poke runs as an
interpreter:

@example
$ poke
[...]
(poke) argv
<stdin>:1:1: error: undefined variable 'argv'
argv;
^~~~
@end example

Accessing the @code{argv} is more than enough for many simple
programs.  However, we may need a more sophisticated handling of
command-line options: support for both short and long style options,
adherence to the GNU coding standards, and the like.  For these cases
poke provides a pickle @code{argp}.  @xref{argp}.

@node Exiting from Scripts
@section Exiting from Scripts

By default a Poke script will communicate a successful status to the
environment, upon exiting:

@example
$ cat hello
#!/usr/bin/poke -L
!#

print "hello\n";
$ ./hello && echo $?
0
@end example

In order to exit with some other status code, most typically to signal
an erroneous situation, the Pokeish way is to raise an @code{E_exit}
exception with the desired exit status code:

@example
raise Exception @{ code = EC_exit, exit_status = 1 @};
@end example

This can be a bit cumbersome to write, so poke provides a more
conventional syntax in the form of an @code{exit} function:

@example
fun exit = (int<32> exit_code = 0) void:
@{
  raise Exception @{ code = EC_exit, exit_status = exit_code @};
@}
@end example

@noindent
Using @code{exit}, the above raise statement becomes the much simpler:

@example
exit (1);
@end example

@node Loading pickles as Modules
@section Loading pickles as Modules

Suppose we want to write a Poke program to extract the contents of
sections in a given ELF object file.  Extracting sections requires
dealing with several data structures encoded in the ELF file, such as
the header, the section header table, the string table (that contains
the names of the sections) and so on.  It would be of course possible
to define Poke types for these structures in the script itself but, as
it happens, GNU poke ships with an already written pickle that
describes the ELF structures.  It is called @file{elf.pk}.

So a script needing to mess with ELF data structures can just make use
of @file{elf.pk} using the load construction:

@example
load elf;
@end example

This looks for a file called @file{elf.pk} in a set of directories,
which are predefined by poke, and loads it.  The list of directories
where poke looks for pickles is stored in the load_path variable as a
colon separated list of directory names, and can be customized:

@example
$ poke
[...]
(poke) load_path
"/home/jemarch/.poke.d:.:/home/jemarch/.local/share/poke:..."
@end example

The default value of @code{load_path} contains both user-specific
directories and system-wide directories.  This assures that all the
pickles installed by poke are available, and that the user can load
her own pickles in her scripts.

Once a pickle is loaded in a script the types, functions and variables
defined in it (either directly or indirectly by loading its own
pickles) become available.

@node elfextractor
@section elfextractor

As an example, in this section we will be hacking a very simple
utility called elfextractor, that extracts the contents of the
sections of an ELF file, whose name is provided as an argument in the
command line, into several output files.  This is the synopsis of the
program:

@example
elfextractor @var{file} [@var{section_name}]
@end example

Where @var{file} is the name of the ELF file from which to extract
sections, and an optional @var{section_name} specifies the name of the
section to extract.

Say we have a file @file{foo.o} and we would like to extract its text
section.  We would use elfextractor like:

@example
$ elfextractor foo.o .text
@end example

Provided @file{foo.o} indeed has a section named @code{.text}, the
utility will create a file @file{foo.o.text} with the section's
contents.  Note how the names of the output files are derived
concatenating the name of the input ELF file and the name of the
extracted section.

If no section name is specified, then all sections are extracted.  For
example:

@example
$ elfextractor foo.o
$ ls foo.o*
foo.o          foo.o.eh_frame       foo.o.shstrtab  foo.o.symtab
foo.o.comment  foo.o.rela.eh_frame  foo.o.strtab    foo.o.text
@end example

This is a possible implementation of elfextractor:

@example
#!/usr/bin/poke -L
!#

/* elfextractor - Extract sections from ELF64 files. */

load elf;

if (!(argv'length in [1,2]))
  @{
    print "Usage: elfextractor FILE [SECTION_NAME]\n";
    exit (1);
  @}

var file_name = argv[0];
var section_name = (argv'length > 1) ? argv[1] : "";

try
  @{
    var fd = open (file_name, IOS_M_RDONLY);
    var elf = Elf64_File @@ fd : 0#B;

    for (shdr in elf.shdr where shdr.sh_type != 0x0)
      @{
        var sname = elf.get_string (shdr.sh_name);

        if (section_name == "" || sname == section_name)
          save :ios elf'ios :file file_name + sname
               :from shdr.sh_offset :size shdr.sh_size;
      @}

    close (fd);
  @}
catch (Exception e)
  @{
    if (e == E_constraint)
      printf ("error: `%s' is not a valid ELF64 file\n", file_name);
    else if (e == E_io)
      printf ("error: couldn't open file `%s'\n", file_name);
    else
      raise e;

    exit (1);
  @}
@end example

First the command line arguments are handled.  The script checks
whether the right number of arguments have been passed (either 1 or 2)
exiting with an error code otherwise.  The file name and the section
name are then extracted from the @code{argv} array.

Once we have the file name and the optional desired section name, it
is time to do the real work.  The code is enclosed in a try-catch
block statement, because some of the operations may result on
exceptions being raised.

First, the ELF file whose name is specified in the command line is
opened for reading:

@example
var fd = open (file_name, IOS_M_RDONLY);
@end example

The built-in function @code{open} returns a file descriptor that can
be subsequently used in mapping operations.  If the provided file name
doesn't identify a file, or if the file can't be read for whatever
reason, an @code{E_io} exception is raised.  Note how the exception is
handled in the @code{catch} block, emitting an appropriate diagnostic
message and exiting with an error status.

Once the ELF file is open for reading, we map an @code{Elf64_File} on
it, at the expected offset (zero bytes from the beginning of the
file):

@example
var elf = Elf64_File @@ fd : 0#B;
@end example

If the file doesn't contain valid ELF data, this map will fail and
raise an @code{E_constraint} exception.  Again, the @code{catch} block
handles this situation.

At this point the variable @code{elf} contains an @code{Elf64_File}.
Since we want to extract the sections contained in the file, we need
to somehow iterate on them.  The section header table is available in
@code{elf.shdr}.  A for-in-where loop is used to iterate on all the
headers, skipping the ``null'' ELF sections which are always empty,
and are characterized by a @code{shdr.sh_type} of 0.  An inner
conditional filters out sections whose name do not match the provided
name in the command line, if it was specified at all.

For each matching section we then save its contents in a file named
after the input ELF file, by calling a function @code{save}, which is
provided by poke:

@example
save :ios elf'ios :file file_name + sname
     :from shdr.sh_offset :size shdr.sh_size;
@end example

The above is exactly what we would have written at the poke REPL!
(modulus trailing semicolon).  How is this supposed to work?  Thing
is, GNU poke commands are implemented as Poke functions.  Let's
consider @code{save}, for example.  It is defined as a function having
the following prototype:

@example
fun save = (int ios = get_ios,
            string file = "",
            off64 from = 0#B,
            off64 size = 0#B,
            int append = 0,
            int verbose = 0) void:
@{ ... @}
@end example

Once a Poke function is defined in the environment, it becomes
available as such.  Therefore, in a poke session we could call it
like:

@example
(poke) save (get_ios, "filename", 0#B, 12#B, 0, 1)
@end example

However, this is cumbersome and error prone.  To begin with, we should
remember the name, position and nature of each argument accepted by
the command.  What is even more annoying, we are forced to provide
explicit values for them, like in the example above we have to pass
the current IOS (the default), and 0 for @code{append} (the default)
just to being able to set @code{verbose}.  Too bad.

To ease commanding poke, the Poke language supports an alternative
syntax to call functions, in which the function arguments are referred
by name, can be given in any order, and can be omitted.  The command
above can be thus written like:

@example
(poke) save :from 0#B :size 12#B :verbose 1
@end example

This syntax is mostly intended to be used interactively, but nothing
prevents to use it in Poke programs and scripts whenever it is deemed
appropriate, like we did in elfextractor.  We could of course have
used the more conventional syntax:

@example
if (section_name == "" || sname == section_name)
  save (elf'ios, file_name + sname,
        shdr.sh_offset, shdr.sh_size, 0, 0);
@end example

@noindent
What style to use is certainly a matter of taste.

Anyhow, once the sections have been written out, the file descriptor
is closed and the program exits with the default status, which is
success.  Should the @code{save} function find any problem saving the
data, such as a full disk, not enough permissions or the like,
exceptions will be raised, caught and maybe handled by our
@code{catch} block.

And this is it!  The complete program is 44 lines long.  This is a
good example that shows how, given a pickle providing a reasonable
description of some binary-oriented format (ELF in this case) poke can
be leveraged to achieve a lot in a very concise way, free from the
many details involved in the encoding, reading and writing of binary
data.

@node Filters
@section Filters

The classic notion of @dfn{filter} in Unix-like systems and other
Operating Systems is of a program that reads certain input using its
standard input, realized some transformations on it, and writes the
result on its standard output.  The filter doesn't need to operate on
all the data at once, nor to buffer it: it reads input in chunks once
it becomes available and is needed, and write output as necessary.

GNU poke provides the so called @dfn{stream IO spaces} in order to
implement binary utilities that act like filters.

@menu
* Stream IO Spaces::      Standard input, output and error streams.
* Reading from Streams::  Reading and flushing input streams.
* Writing to Streams::    Writing and flushing output streams.
* pk-strings::            Putting everything together with a simple example.
@end menu

@node Stream IO Spaces
@subsection Stream IO Spaces

GNU poke provides support for opening and operating on several kinds
of IO spaces: files, memory buffers, process memory, @i{etc}.  An IO
space can be opened for reading, for writing or both, and they all can
be accessed randomly, @i{i.e.} to read or write some data from/to a
particular offset in the IO space.

Writing a filter would involve to peek data from the standard input
and then poke the data (maybe transformed) to the standard output or
the standard error output.  However, we know that these devices are
not random oriented devices that can be accessed at any offset, but
rather stream-like devices whose special characteristics are:

@itemize @minus
@item
They are either read-only or write-only.  You cannot write to an input
stream and you cannot read from an output stream.
@item
Once data is read from an input stream, it can't be read again.  It is
not buffered.
@item
Once data is written to an output stream, it can't be altered again.
@end itemize

@noindent
Given the above characteristics of streams like the standard input and
output, it is clear we cannot use the File IO streams on them.  It is
also not clear how could we perform a map operation on a stream
device: in poke these operations always require an explicit offset.

GNU poke solves this by providing three special IO spaces that the
user can open using the following handlers:

@table @code
@item <stdin>
Read-only stream IOS that is connected to the standard input of the
poke process.
@item <stdout>
Write-only stream IOS that is connected to the standard output of the
poke process.
@item <stderr>
Write-only stream IOS that is connected to the standard error of the
poke process.
@end table

@noindent
Opening these IO spaces is done exactly like with any other IO space.
Example:

@example
var stdin = open ("<stdin>");
@end example

@node Reading from Streams
@subsection Reading from Streams

Unlike normal Unix file descriptors, read-only poke streams are
buffered.  Let's supposed we open the standard input:

@example
var stdin = open ("<stdin>");
@end example

@noindent
Then we map some data (one byte) at some particular offset:

@example
var b = byte @@ stdin : 28#B;
@end example

@noindent
The mapping operation above causes poke to read 28 bytes from the
standard input and to set the variable @code{b} to the 28-th byte.
The bytes read from the stream are still available (buffered) and the
offsets are still relative from the ``beginning'' of the standard
input:

@example
var c = byte @@stdin : 0#B;
@end example

@noindent
The byte in @code{c} is the byte that appeared in the standard input
27 bytes before the byte in @code{b}.

This buffering is nice, and it is what allows us to access the
standard input like if it were a random oriented device.  However, it
is obvious we would be in trouble if we filter big amounts of data,
like in a network interface: we would likely use all available memory.

To allow filtering big amount of data, poke allows to @dfn{flush} the
read-only streams.  Flushing means that the buffered data in the
read-only stream is ``forgotten'', and trying to access it will result
in an exception:

@example
var stdin = open ("<stdin>");

var b = byte @@ stdin : 28#B;
var c = byte @@ stdin : 29#B;
flush (stdin, 29#B);
var d = byte @@ stdin : 30#B;
var e = byte @@ stdin : 20#B; /* Exception.  */
@end example

@node Writing to Streams
@subsection Writing to Streams

Similarly, unlike normal Unix file descriptors, write-only poke
streams are also buffered and can be accesses randomly:

@example
var stdout = open ("<stdout>");
@end example

@noindent
We can write to the output stream by mapping as usual:

@example
byte @@ stdout : 28#B = 0xff;
@end example

@noindent
The mapping operation above writes 27 null bytes, @i{i.e.} it fills
until reaching the requested offset in the output stream.  Then it
writes the byte @code{0xff} at offset 28 bytes.  If we then write to
one of the buffered bytes:

@example
byte @@ stdout : 27#B = 0xab;
@end example

@noindent
This results in updating the byte that will be written right before
the @code{0xff} once the output stream is flushed.

Again, we cannot buffer ad-infinitum: we would exhaust all available
output.  Therefore,  XXX

@node pk-strings
@subsection pk-strings

Below you can find a very simple Poke program that works like the
standard Unix utility @command{strings}.

@example
#!/usr/local/bin/poke -L
!#

/* Printable ASCII characters: 0x20..0x7e */

var stdin = open ("<stdin>");
var stdout = open ("<stdout>");

var offset = 0#B;

try
@{
  flush (stdin, offset);

  var b = byte @@ stdin : offset;
  if (b >= 0x20 && b <= 0x7e)
    byte @ stdout : iosize (stdout) = b;

  offset = offset + 1#B;
@}
until E_eof;

close (stdin);
close (stdout);
@end example

@node Configuration
@chapter Configuration

@menu
* pokerc::			User's initialization file.
* Load Path::			Determining location of modules.
* Styling::			Changing the appearance of poke's output.
@end menu

@node pokerc
@section @file{.pokerc}
@cindex @file{.pokerc}

Upon invocation poke will read and execute the commands of an
initialization file, if it exists: the @file{pokerc} file.  There are
two ways to keep an initialization file.

The simple, traditional way is to have a file named @file{.pokerc} in
your home directory.  GNU poke will look for a file like that first.

If @file{.pokerc} is not found your home directory, poke looks in the
locations specified by the @dfn{XDG Base Directory
Specification}@footnote{https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html}
for a file named @file{poke/pokerc.conf}.  For example, you could use
@file{~/.config/poke/pokerc.conf}.

Which way is better depends on your specific requirements and taste.

GNU poke can be insturcted to not read the initialization file by
passing @code{-q} or @code{--no-init-file} in the command line.

Example of initialization file:

@example
# My poke configuration.
.set endian host
.set obase 16
.set pretty-print yes
pk_dump_cluster_by = 4
.load ~/.poke.d/mypickles.pk
@end example

@node Load Path
@section Load Path
@cindex load path

The @code{load_path} Poke variable contains a list of directories
separated by the colon character (@code{:}).  When a module is loaded
using the @code{load} construction, these directories are searched in
sequential order for the file corresponding to the requested module.

Empty directory names and entries that do not name existing
directories are ignored.

Some entries have special meanings:

@table @code
@item %DATADIR%
This is interpreted as the system-wide datadir directory, that depends
on the prefix where poke is installed.
@end table

For example, say you want to maintain @file{.pk} files in your
@file{~/.poke.d} directory.  You will probably want to add that
directory to the @code{load_path}, when poke initializes.  A way to do
that is to add a command like this to your @file{pokerc} file:

@example
load_path = getenv ("HOME") + "/.poke.d:" + load_path
@end example

If the environment variable @code{POKE_LOAD_PATH} is defined in the
environment, its value is added to @code{load_path} at poke startup
time.

@node Styling
@section Styling
@cindex styled output

XXX

@node IOS
@chapter IOS

The @file{ios} pickle contains functions that are useful to work with
IO spaces.

@menu
* ios_dump_bytes::              Dumping bytes on the terminal.
* ios_copy_bytes::              Copying bytes between IO spaces.
* ios_save_bytes::              Saving a range of bytes to a file.
@end menu

@node ios_dump_bytes
@section ios_dump_bytes

The @code{ios_dump_bytes} function prints a byte dump of an area in a
given IO space.

It has the following prototype:

@example
fun ios_dump_bytes = (int<32> @var{ios},
                      offset<uint<64>,b> @var{from},
                      offset<uint<64>,b> @var{size},
                      offset<uint<64>, b> @var{group_by} = 2#B,
                      int<32> @var{cluster_by} = 8,
                      int<32> @var{ruler} = 0,
                      int<32> @var{ascii} = 0,
                      string @var{unknown_byte} = "??",
                      uint<8> @var{nonprintable_char} = '.') void
@end example

@noindent
Where @var{ios} is the IO space form which dump bytes.

@var{from} is the offset from which start dumping bytes. This location
is truncated to bytes.

@var{size} is an offset specifying the amount of bytes to dump.  The
offset is rounded up to the next byte.

@var{group_by} determines how the bytes are grouped together in the
output.  Defaults to two bytes.

@var{cluster_by} specifies to display additioanl space after the
specified number of groups have been displayed.  Defaults to 8.

@var{ruler} is a boolean determining whether to include a ruler line
in the output.  Defaults to 0.

@var{ascii_p} is a boolean determining whether to include an ASCII
dump.  Defaults to 0.

@var{unknown_byte} is the string to use to denote bytes that can't be
read from the IO space.  Defaults to @code{"??"}.  Note this string is
expected to be of length two characters.

@var{nonprintable_char} is the character code to use to denote
non-printable characters.  Defaults to @code{'.'}.

This function may raise @code{E_io}, @code{E_perm}, and @code{E_eof}
on several error conditions.

@node ios_copy_bytes
@section ios_copy_bytes

The @code{ios_copy_bytes} copies a range of bytes between IO spaces.

It has the following prototype:

@example
fun ios_copy_bytes = (int<32> @var{from_ios}, int<32> @var{to_ios},
                      offset<uint<64>,B> from,
                      offset<uint<64>,B> to,
                      offset<uint<64>,B> size) void
@end example

@noindent
Where @var{from_ios} is the origin IO space, @i{i.e.} the IO space
from which bytes are to be copied.

@var{to_ios} is the destination IO space, @i{i.e.} the IO space to
which bytes are to be copied.

@var{from} is a byte offset in @var{from_ios} with the beginning of
the range to copy.

@var{to} is a byte offset in @var{to_ios} where the bytes will be
copied.

@var{size} is a byte offset specifying the amount of data to be
copied.

@node ios_save_bytes
@section ios_save_bytes

The function @code{ios_save_bytes} saves a range of bytes from an IO
space to a file.

It has the following prototype:

@example
fun ios_save_bytes = (int<32> ios, string file,
                      offset<uint<64>,B> from,
                      offset<uint<64>,B> size,
                      int<32> append_p) void
@end example

@noindent
Where @var{ios} is the IO space from which save bytes.

@var{file} is the name of the file where to save the bytes.

@var{to} and @var{size} are byte offsets in the IO space determining
the range of bytes to save.

By default the output file is truncated if it already exists.  If
@var{append_p} is true, then the data is instead appended at the end
of the file.

@node Time
@chapter Time

Systems encode time-stamps or dates in several ways.  GNU poke
provides support for some of them.

@menu
* POSIX Time::			The @file{time} pickle.
@end menu

@node POSIX Time
@section POSIX Time

In POSIX systems time is encoded as a certain number of seconds
elapsed since some origin date, namely the first of January 1970.  The
@file{time} pickle provides two types for 32-bit and 64-bit POSIX time
encodings:

@example
POSIX_Time32
POSIX_Time64
@end example

A function @code{ptime} is also provided, that prints a human-readable
representation of a POSIX date, given a number of seconds.

@node Colors
@chapter Colors

Colors are often found in binary data, encoded in several different
ways.  GNU poke provides several pickles that makes it easier to work
with these colors.

@menu
* The Color Registry::		The @file{color} pickle.
* RGB24 Encoding::		Encoding colors with three bytes.
@end menu

@node The Color Registry
@section The Color Registry

The @file{color} pickle provides a registry of @dfn{standard colors},
organized as a space of integers.  Each integer identifies a standard
color, which are accessible as variables named @code{color_*} after
the pickle is loaded.  Example:

@example
(poke) load color
(poke) color_tomato
114
@end example

The purpose of having this register is to have a global namespace for
colors that can be used in different pickles.  The position of each
color in the registry is totally unrelated to how the color may be
encoded.  Other pickles, as we shall see below, contain tables
associating standard colors with their encoding, such as RGB.

If you want to add a new color that is not in the standard collection,
you can use the @code{color_register} function, which gets no
arguments:

@example
(poke) var mycolor = color_register
(poke) mycolor
490
@end example

The total number of registered colors is recorded in the variable
@code{color_num_colors}.  You can use it to iterate on all the colors
in the register, from @code{0} to @code{color_num_colors - 1}.

The index in the registry of the first user-defined color is hold in
the variable @code{color_LAST}.   For example, if you wanted to store
a frob per standard color, you would do it like:

@example
(poke) type Frobs = Frob[color_LAST]
@end example

The pickle also provides a function @code{color_name} that, given a
color code, returns a printable name for the color, @i{i.e.} a string
describing it.  For user defined colors, this string is fixed:

@example
(poke) color_name (23)
"lavender blush"
(poke) color_name (color_register)
"user-defined color"
@end example

@noindent
If a non-existent color code is passed to @code{color_name} the
function raises @code{E_out_of_bounds}:

@example
(poke) color_name (color_num_colors)
unhandled out of bounds exception
@end example

@node RGB24 Encoding
@section RGB24 Encoding

The @dfn{RGB24} encoding encodes each color as a triplet of @dfn{color
beans}, each beam indicating a level of red, green and blue
respectively.

Types are provided for both the color beams, and the triplets:

@example
type RGB24_Color_Beam = uint<8>;
type RGB24_Color = RGB24_Color_Beam[3];
@end example

@noindent
The indexes @code{RGB24_RED}, @code{RGB24_GREEN} and @code{RGB24_BLUE}
can be used to access to a specific beam of a given color:

@example
(poke) rgb24_color[color_tomato][RGB24_GREEN]
99UB
@end example

The @file{rgb24} pickle also provides a table associating poke
standard colors (@pxref{The Color Registry}) with their RGB24
encodings.  Once the pickle is loaded, this table is available in the
variable @code{rgb24_color}.  The table is to be indexed by color
codes:

@example
(poke) load rgb24
(poke) rgb24_color[color_tomato]
[255UB,99UB,71UB]
@end example

@node Audio
@chapter Audio

@menu
* MP3::				Editing MP3 files.
@end menu

@node MP3
@section MP3
@cindex MP3

@menu
* ID3V1 Tags::			The @file{id3v1} pickle.
@end menu

@node ID3V1 Tags
@subsection ID3V1 Tags

The @file{id3v1} pickle provides abstractions in order to edit the
metadata stored in a MP3 file.

@subsubsection Song Genres

The ID3V1 tags support the notion of @dfn{song genre}.  The space for
genres is from 0 to 254.  The genre code 255 is reserved to mean ``no
genre''.

The table @code{id3v1_genres} can be indexed with a code in order to
get the corresponding genre name:

@example
(poke) id3v1_genres[14]
"rhythm and blues"
@end example

Conversely, the function @code{id3v1_search_genre} gives us the code
of a genre, given its name.  The prototype of this function is:

@example
fun id3v1_search_genre = (string name) uint<8>:
@end example

@noindent
For example:

@example
(poke) id3v1_search_genre ("rock")
17UB
@end example

If @code{id3v1_search_genre} is given a name that doesn't correspond
with any genre in the genres table, then @code{E_inval} is raised.

@subsubsection The ID3V1_Tag Type

The main data structure defined in the @file{id3v1} pickle is
@code{ID3V1_Tag}, which corresponds to a ID3V1 tag (surprise!).

Tags comprise the following information:

@itemize @minus
@item The title of the song, which is limited to 30 bytes.
@item The artist, which is limited to 30 bytes.
@item The album, which is limited to 30 bytes.
@item The year, which is encoded as text in 4 bytes, each byte
containing the ASCII code for the corresponding digit.
@item A comment, which is limited to either 28 or 30 bytes, depending
whether the tag contains track information or not.
@item An optional track, which is an unsigned 8-bit number.
@end itemize

The specification does not mention any specific encoding for the
entries that store text (such as title or artist), but it is safe to
assume some ASCII-compatible encoding is used.

The text entries are stored as arrays of characters, and they are
@emph{not} finished by NULL characters.  Instead the arrays of
characters are filled with whitespaces (ASCII code 0x20) at the
right.  For example, the artist name Picasso encoded in ID3V1 would
be:

@example
['P','i','c','a','s','s','o',' ',' ', ..., ' ']
@end example

In order to ease the manipulation of the text fields, setters and
getters are provided in order to handle these values as strings and
not as whitespace-filled arrays of characters.  Example:

@example
(poke) tag.title
[48UB,49UB,32UB,45UB,32UB,...]
(poke) tag.get_title
"01 - Eclipse De Mar"
@end example

@noindent
Also for setters:

@example
(poke) tag.set_title ("Join us Now")
(poke) tag.title
[74UB,111UB,105UB,110UB,32UB,...]
@end example

Setters and getters are also provided in order to manipulate the year
as an integer value:

@example
(poke) tag.year
[0x31UB,0x39UB,0x38UB,0x30UB]
(poke) tag.get_year
1980
(poke) tag.set_year (1988)
@end example

@node Object Formats
@chapter Object Formats

This chapter is of special interest to toolchain developers, people
doing reverse-engineering, and similar low-level development
activities.

@menu
* ELF::                         Editing ELF files.
* Dwarf::	                Editing DWARF data.
@end menu

@node ELF
@section ELF

XXX

@node Dwarf
@section Dwarf

XXX

@node Programs
@chapter Programs

This chapter describes pickles whose purpose is to facilitate the
writing of binary utilities in Poke.

@menu
* argp::			Handling of command-line options.
@end menu

@node argp
@section argp

Like any other program, Poke scripts often need to handle options
passed in the command line.  The @code{argp} pickle provides a
convenient and easy way to handle these arguments.

The main entry point of the pickle is the function @code{argp_parse},
with the following prototype:

@example
fun argp_parse = (string @var{program},
                  string @var{version} = "",
                  string @var{summary} = "",
                  Argp_Option[] @var{opts} = Argp_Option[](),
                  string[] @var{argv} = string[](),
                  int @var{allow_unknown} = 0) string[]
@end example

@noindent
Where @var{program} is the name of the program providing the
command-line options.  @var{version} is either a string identifying
the version of the program, or the empty string. @var{summary} is a
short summary of what the program does.  @var{opts} is an array of
@code{Argp_Option} structs, each of which describe a command line
option and, in particular, a handler for that option.  @var{argv} is
an array of string containing the command line elements to process.
Finally, @var{allow_unknown} specifies whether unknown options
constitute an error or not.  This last option is useful in order to
support sub-parsers.

Once invoked, @code{argp_parse} analyzes the command line provided in
@var{argv}, recognizes options, performs checks making sure that all
specified options are well formed, and that every option requiring an
argument gets one, invokes the handlers registered for each option,
and finally returns an array of non-option arguments in @var{argv},
for further processing by the user.

Each @code{Argp_Option} struct describes an option.  They have this
form:

@example
type Argp_Option =
  struct
  @{
    string @var{name};
    string @var{long_name};
    string @var{summary};
    string @var{arg_name};
    int @var{arg_required};
    Argp_Option_Handler @var{handler};
  @};
@end example

@noindent
Where @var{name} is a string of size one character specifying the
short name for the option. @var{long_name} is a non-empty string of
any size specifying the long name for the option. @var{summary} is a
short paragraph with a summary of that the option does.
@var{arg_name}, if specified, is a string to be used to describe the
argument of the option in the @code{--help} output.  If @var{arg_name}
is not specified then @code{"ARG"} is used.  @var{arg_required} is a
boolean specifying whether the option accepts and expects an
argument.  Defaults to 0.  Finally, @var{handler} is a function that
will be invoked to process the option.  The function has the following
prototype:

@example
type Argp_Option_Handler = (string)void
@end example

The @code{argp_parse} function provides default handlers for several
standard options:

@table @command
@item --help
The default handler for this option prints out an usage message in the
standard GNU way, and exits.  In particular, this output is compatible
with the @command{help2man} utility.
@item --version
The default handler for this option prints out the name and the
version of the program, and exits.
@end table

@noindent
If the user provides handlers for the options above, these take
precedence wrt the default handlers.

Several short options can be accumulated this way:

@example
foo -qyz
@end example

The special option @code{--}, if found in the command line, stops the
processing of options.  Everything found about it is considered
non-option arguments.

@node poked
@chapter poked

The @code{poke} application is a REPL (Read-Evaluate-Print-Loop) and
therefore the user interacts with it by providing Poke statements or
expressions (or dot-commands) in the standard input, and then the
program prints the response to the commands in the standard output, or
standard error.  This is good for human users, but not that convenient
for other programs that may wish to communicate with a poke incremental
compiler.

@code{poked} is a different sort of application that accepts Poke code
(and more) over Unix sockets and provides responses over the same
socket.  This is very convenient for programs to interact with the
poke incremental compiler.  The client programs that interact with
@code{poked} are called @emph{pokelets}.

This form of interaction is particularly userful to write user
interfaces to GNU poke, in the form of several pokelets.  Examples
of such interfaces are
@url{https://git.savannah.gnu.org/cgit/poke/poke-el.git,Emacs
interface to poke} and @url{https://sourceware.org/pacme/,pacme}.

@code{poked} is basically a message broker that provides (for now)
120 input channels and 120 output channels.  The first 60 channels
are reserved for the @code{poked} developers in order to provide
standard channels whose meaning is fixed and the same for any pokelet.
The rest of the channels can be freely dedicated for any purpose by
anybode else.

The combination of @emph{channel number} and @emph{channel direction}
creates the client's @emph{role}.  For now, it's a single byte.  The
most significant bit determines whether it's an input or output
channel.  Or in poke syntax:

@example
var PDAP_DIRECTION_IN  = 0 as uint<1>,
    PDAP_DIRECTION_OUT = 1 as uint<1>;

type PDAP_Role = struct uint<8>
  @{
    uint<1> direction;
    uint<7> channel : 0 < channel && channel <= 120;
  @};
@end example

Pokelets are expected to write their role over the socket when
they're connected to the @code{poked}.

@code{poked}, on start up, will create a Unix socket and prints
current options to the @code{stdout}.  Options are a pair of
space-separated key/values, each on one line.  Options will end
with an empty line.  An example of @code{poked} output:

@example
socket_path /tmp/poked.ipc
pdap_version 0

@end example

Currently, only first two input channels are active.  Input channel
number 1 is for @emph{code input} which means anything sent to this
channel will be compiled and executed using @code{pk_compile_buffer}
function of @code{libpoke}; and input channel number 2 is for
@emph{command input} which will compile and execute the input message
using @code{pk_compile_statement} function.

For output channels:

@table @dfn
@item Channel 1
Terminal output.
@item Channel 2
View (vu) output.
@item Channel 3
Poke Virtual Machine (PVM) disassembly output.
@item Channel 4
Treevu (tree view) output (not supported yet).
@item Channel 5
Auto-completion output.
@item Channel 6
CPU disassembly output.
@end table

The format of each message written to output channels are documented
in @code{pdap.pk} pickle.

The following functions are available for pokelets:

@table @code
@item poked_pdap_version
Version of current PDAP (PokeD Application Protocol).
@item poked_libpoke_version
Version of @code{libpoke}.
@item poked_restart
Restart the Poke environment of @code{poked}.
@item poked_exit
Exit the @code{poked}.
@item poked_chan_send
Signature: @code{(uint<7> chan, byte[] data) void}.
Send data to the specified output channel.
@end table

The following variable is also available for pokelets to use:

@table @code
@item poked_after_cmd_hook
An array of callbacks to be run after executing the @emph{command}
(received from input channel 2).  Callback signature: @code{()void}.
@end table

@node Programming Emacs Modes
@chapter Programming Emacs Modes

GNU poke ships with a bunch of Emacs major modes that eases writing
Poke and PVM assembly.  This chapter documents these modes.

@menu
* poke-mode::                   Major mode for writing Poke programs.
* poke-map-mode::               Major mode for writing map-files.
* poke-ras-mode::               Major mode for writing RAS programs.
@end menu

@node poke-mode
@section poke-mode

@code{poke-mode} is a major mode for editing Poke source files,
@i{i.e.} @file{.pk} files.  It provides font-lock, auto-completion and
indentation features.

@node poke-map-mode
@section poke-map-mode

@code{poke-map-mode} is a major mode for editing poke map-files.  It
provides font-lock.  @xref{Maps and Map-files}.

@node poke-ras-mode
@section poke-ras-mode

@code{poke-ras-mode} is a major mode for editing PVM assembly.  It is
used in the development of GNU poke.  See the file @code{libpoke/ras}
in the source distribution for more information about RAS, our
Retarded Assembler.

@node Vim Syntax Highlighting
@chapter Vim Syntax Highlighting

GNU poke ships with a vim syntax highlighter for *.pk files.
Normally this should be installed as part of your existing poke installation
and work out of the box.
If that is not the case this chapter explains how to install it manually.

@menu
* @file{poke.vim}::    Poke syntax highlighter
@end menu

@node @file{poke.vim}
@section @file{poke.vim}

@code{etc/vim/syntax/poke.vim} is a syntax highlighter for Poke source
@i{i.e.} @file{.pk} files. To use it, first place the file in your
@code{~/.vimrc/syntax/} folder on Unix based systems, or in
@code{$HOME/vimfiles/syntax/} on Windows. There are a few ways to tell
Vim to use @code{poke.vim} for @file{.pk} files, but the quickest is
to add the following line to your @code{.vimrc}:

@example
au BufRead,BufNewFile *.pk set filetype=poke
@end example

@node Dot-Commands
@chapter Dot-Commands

@menu
* load command::		Loading pickles.
* source command::              Executing commands in files.
* file command::		Opening and selecting file IO spaces.
* mem command::			Opening and selecting memory IO spaces.
* nbd command::			Opening and selecting NBD IO spaces.
* proc command::                Opening and selecting process IO spaces.
* sub command::                 Opening IO sub-spaces.
* ios command::			Switching between IO spaces.
* close command::		Closing IO spaces.
* doc command::                 Online manual.
* editor command::		Using an external editor for input.
* info command::		Getting information about open files, @i{etc}.
* set command::			Querying and setting global options.
* vm command::			Poke Virtual Machine services.
* exit command::		Exiting poke :(
* quit command::                Likewise.
@end menu

@node load command
@section @code{.load}
@cindex @code{.load}

The @command{.load} command loads a file containing Poke code and
compiles and executes it.  These files usually have the extension
@file{.pk}.

If a relative path is provided, then @file{@var{prefix}/share/poke}
is tried first as a base directory to find the specified file.  If it
is not found, then the current directory is tried next.

If the environment variable @code{POKEDATADIR} is defined, it replaces
@file{@var{prefix}/share/poke}.  This is mainly intended to test a
poke program before it gets installed in its final location.

If an absolute path is provided, it is used as-is.

@node source command
@section @code{.source}
@cindex @code{.source}
@cindex executing command files
The @command{.source} command executes command lines from some given
file.  The syntax is:

@example
.source @var{path}
@end example

@noindent
where @var{path} is a path to the file containing the commands.
@xref{Command Files}.

@node file command
@section @code{.file}
@cindex @code{.file}
@cindex opening files
@cindex IO space
The @command{.file} command opens a new IO space backed by a file, or
switches to a previously opened file.  The syntax is:

@example
.file @var{path}
@end example

@cindex tags, file ID tags
@noindent
where @var{path} is a path to a file to open, which can be relative to
poke's current working directory or absolute.

Tilde expansion is performed in @var{path}, much like it's done in the
shell.  This means you can include special characters like @code{~}
(which will expand to your home directory) delimit the file name with
@code{"} in case it includes leading or trailing blank characters,
@i{etc}.

When a new file is opened it becomes the current IO space.  From that
point on, every map executed in the REPL or while loading a Poke
program will operate on that IO space:

@example
(poke) .file foo.o
The current file is now `foo.o'.
@end example

When @command{.file} opens a file, it is opened in whatever mode makes
more sense: if the file allows to be written and read then the IO
space is open in read/write mode, for example.

If the specified file path doesn't exist then poke emits an error.
However, the flag @command{/c} ( for ``create'') can be passed to the
command to tell poke it should create a new file.

@node mem command
@section @code{.mem}
@cindex @code{.mem}
@cindex opening memory buffers
@cindex IO space
The @command{.mem} command opens a new IO space backed by a memory
buffer.  The syntax is:

@example
.mem @var{name}
@end example

@cindex tags, file ID tags
@noindent
where @var{name} is the name of the buffer to create.  Note that poke
adds prefix and trailing asterisk characters, to differentiate file
names from buffer names.

When invoked with no @var{name}, poke generates a new unique name for
the buffer.

When a new memory buffer IOS is opened it becomes the current IO
space.  @xref{file command}.

@node nbd command
@section @code{.nbd}
@cindex @code{.nbd}
@cindex opening NBD buffers
@cindex IO space
The @command{.nbd} command opens a new IO space backed by an external
NBD server.  The syntax is:

@example
.nbd @var{uri}
@end example

@cindex tags, file ID tags
@noindent
where @var{uri} is the name of the newly created buffer, matching the
@url{https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md,
NBD URI specification}.

When a new NBD IOS is opened, it becomes the current IO
space.  @xref{file command}.

NBD support in GNU poke is optional, depending on whether poke was
compiled against @url{http://libguestfs.org/libnbd.3.html,, libnbd}.

For an example of connecting to the guest-visible content of a qcow2
image, with the default export name as exposed by using qemu as an NBD
server:

@example
$ qemu-nbd --socket=/tmp/mysock -f qcow2 image.qcow2
$ poke
(poke) .nbd nbd+unix:///socket=?/tmp/mysock
The current file is now `nbd+unix:///socket=?/tmp/mysock'.
@end example

@node proc command
@section @code{.proc}
@cindex @code{.proc}

The @command{.proc} command opens a new IO space backed by the memory
mapped for some live process, or switches to a previously opened
process.  The syntax is:

@example
.proc @var{pid}
@end example

@noindent
where @var{pid} is a process identifier.  In most systems this is a
number.

@node sub command
@section @code{.sub}
@cindex @code{.sub}

The @command{.sub} command opens a new IO sub-space.  The syntax is:

@example
.sub @var{ios}, @var{base}, @var{size}[, @var{name}]
@end example

@noindent
where @var{ios} is the tag of some existing IO space, @var{base} and
@var{size} are integers denoting the base and size of the range
covered by the sub-space (in bytes) and @var{name} is an optional
name.

@node ios command
@section @code{.ios}
@cindex @code{.ios}

@cindex tags, file ID tags
A list of open files, and their corresponding tags, can be obtained
using the @command{.info ios} command.  Once a tag is known, you can
use the @command{.ios} command to switch back to that file:

@example
(poke) .ios #1
The current IOS is now `foo.o'.
@end example

@node close command
@section @code{.close}
@cindex @code{.close}

@cindex IO space
The @command{.close} command closes the selected IO space.  The syntax
is:

@example
.close @var{#tag}
@end example

@noindent
where @var{#tag} is a tag identifying an open IO space.

Note that closing an IO space with this dot-command implies closing
any sub IO space having it as a base.

@node doc command
@section @code{.doc}
@cindex @code{.doc}
@cindex doc
The @command{.doc} command is used to display this manual in poke's REPL.
The syntax is:

@example
.doc [@var{node}]
@end example

@noindent
where @var{node} is an optional parameter which indicates the chapter
or section at which the manual should be opened.

This command uses whatever documentation viewer configured using
@command{.set doc-viewer} whose valid options are either
@command{info} and @command{less}.

Unless @command{doc-viewer} is set to @command{less}, this command
uses the info program (@ref{Top,,, info, The GNU Texinfo Manual}) to
interactively present the manual.  If info is not installed, then less
is tried next.

When using less to display the documentation, the entry in the Table
of Contents corresponding to the requested entry is highlighted.  Just
press @key{/} and then @key{RET} to jump to the corresponding section
in the manual.

If neither info nor less are installed, the @command{.doc} command
will fail.  If poke is not running interactively then @command{.doc}
does nothing.


@node editor command
@section @code{.editor}
@cindex @code{.editor}
@cindex editor
The @command{.editor} command (usually abbreviated as @command{.edit})
invokes an external text editor on a temporary file.  You can then
put contents on that file, save it and exit the editor.  At that point
poke will read the file contents, turn them into a single line and
execute them in the repl.
If poke is not running interactively, then @command{.editor} does nothing.

The editor used is identified by the @code{EDITOR} environment
variable.

@node info command
@section @code{.info}
@cindex @code{.info}

The @command{.info} command provides information about several kinds
of entities.  The recognized sub commands are:

@table @command
@item .info ios
Display a list of open files.

@example
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
  #1	FILE	r	0x00000000#B	0x0000df78#B	foo.o
* #0	FILE	rw	0x00000000#B	0x00000022#B	foo.bson
@end example

@cindex IO space
The file acting as the current IO space is marked with an asterisk
character @code{*} at the beginning of the file.  The mode in which
the file is open is also specified.  The @code{Id} field is the tag of
the file that can be passed to the @command{.file} command in order to
switch to it as the new current IO space:

@example
(poke) .ios #1
The current file is now `foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* #1	FILE	r	0x00000000#B	0x0000df78#B	foo.o
  #0	FILE	rw	0x00000000#B	0x00000022#B	foo.bson
@end example

@item .info variables [@var{regexp}]
@cindex variables
Shows a list of defined variables along with their current values and
the location where the variables were defined. If a regular expression
is provided then only the types whose name match the expression are
listed.  If a regular expression is provided then only the types whose
name match the expression are listed.
@item .info functions [@var{regexp}]
Shows a list of defined functions along with their prototypes and the
location where the functions were defined.  If a regular expression is
provided then only the types whose name match the expression are
listed.  If a regular expression is provided then only the types whose
name match the expression are listed.
@item .info types [@var{regexp}]
Shows a list of defined types along with the locations where the types
were defined.  If a regular expression is provided then only the types
whose name match the expression are listed.
@item .info type @var{name}
Prints a description of the type with name @var{name}.
@end table

@node set command
@section @code{.set}
@cindex @code{.set}

@cindex global settings
The @command{.set} command allows you to inspect and set the value of
global settings.  The following invocations are valid:

@table @code
@item .set
Prints all settings along with their values.
@item .set @var{setting}
Prints the value of the given @var{setting}.
@item .set @var{setting} @var{value}
Sets the value of @var{setting} to @var{value}.
@end table

@noindent
The following settings can be handled with @command{.set}:

@include poke-settings.texi

@node vm command
@section @code{.vm}
@cindex @code{.vm}

@cindex virtual machine
The Poke Virtual Machine (PVM) executes the programs that are the
result of the compilation of what you write in the REPL or the pickles
you load.  The @command{.vm} command provides sub-commands to interact
with the PVM.

@menu
* @:.vm disassemble::		PVM and native disassembler.
* @:.vm profile::               Profiling Poke programs.
* @:.vm dispatch::              PVM dispatching strategy.
@end menu

@node @:.vm disassemble
@subsection @code{.vm disassemble}
@cindex disassembler
The @command{.vm disassemble} command provides access to the PVM
disassembler.  It supports the following subcommands:

@table @command
@item .vm disassemble expression @var{expr}
Dumps the assembler corresponding to the Poke expression @var{expr}.
@item .vm disassemble function @var{function}
Dumps the assembler corresponding to the Poke function called
@var{function}.  The function shall be reachable from the top-level.
@item .vm disassemble mapper @var{expr}
If @var{expr} is a mapped value, dumps the assembler corresponding to
its mapper function.
@item .vm disassemble writer @var{expr}
If @var{expr} is a mapped value, dumps the assembler corresponding to
its writer function.
@end table

The disassembler will provide a PVM disassembly by default, but it can
be passed the flag @command{/n} to do a native disassembly instead in
whatever architecture running poke.

@node @:.vm profile
@subsection @code{.vm profile}
@cindex profiler

The @command{.vm profile} command provides access to the PVM profiler.
This profiler is only available if poke has been configured with PVM
profiling support.  This command supports the following subcommands:

@table @command
@item .vm profile reset
Resets the profiling counts in the virtual machine.
@item .vm profile show
Outputs a summary with both counts and sample information.
@end table

@node @:.vm dispatch
@subsection @code{.vm dispatch}
@cindex dispatch

The @command{.vm dispatch} command prints the current dispatching
strategy in PVM (provided by Jitter).  There are four different kind
of dispatches:

@itemize
@item switch
@item direct threading
@item minimal threading
@item no threading
@end itemize

@code{no threading} is the fastest one and the least portable
dispatch, and @code{switch} is the slowest and the most portable
one.

Jitter by default will use the most efficient dispatch which is
both stable and available for the current configuration.

@node exit command
@section @code{.exit}
@cindex @code{.exit}
@cindex quitting
@cindex @code{exit}
The @code{.exit} command exits poke.  The syntax is:

@example
.exit [@var{status}]
@end example

@noindent
Poke will terminate, returning the exit status @var{status}.
If @var{status} is omitted, then the exit status zero will be returned.

@node quit command
@section @code{.quit}
@cindex @code{.quit}
@cindex quitting
@cindex @code{quit}

The @code{.quit} command behaves exactly like @code{.exit}.
@xref{exit command}.

@node Commands
@chapter Commands

@menu
* dump::			Binary dumps.
* copy::			Copying data around.
* save::			Save data into a file.
* sdiff::                       Comparing mapped values byte by byte.
* extract::			Extract contents of values to memory IOS.
* scrabble::			Scrabble memory chunks based on patterns.
@end menu

@node dump
@section @command{dump}
@cindex @command{dump}

At the most basic level, memory can be examined byte by byte.
To do this, use the @command{dump} command.

This command has the following synopsis.

@example
dump [:from @var{offset}] [:size @var{offset}] [:ios @var{int}] [:ruler @var{bool}]
     [:ascii @var{bool}] [:group_by @var{int}] [:cluster_by @var{int}]
@end example

@noindent
All arguments are optional, which means the simplest use of the command is
to simply type @command{dump}:

@example
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 9b07 5a61 4783 f306 4897 f37c fe39 4cd3  ..ZaG...H..|.9L.
00000010: b6a2 a578 8d82 7b7f 2076 374c 3eab 7150  ...x..@{. v7L>.qP
00000020: 31df 8ecb 3d33 ee12 429b 2e13 670d 948e  1...=3..B...g...
00000030: 86f1 2228 ae07 d95c 9884 cf0a d1a8 072e  .."(...\........
00000040: f93c 5368 9617 6c96 3d61 7b92 9038 a93b  .<Sh..l.=a@{..8.;
00000050: 3b0d f8c9 efbd a959 88d0 e523 fd3b b029  ;......Y...#.;.)
00000060: e2eb 51d5 cb5b 5ba9 b890 9d7a 2746 72ad  ..Q..[[....z'Fr.
00000070: 6cbd 6e27 1c7f a554 8d2e 77f9 315a 4415  l.n'...T..w.1ZD.
@end example


@noindent
The first row is the @dfn{ruler} which serves as a heading for each
subsequent row.   On the left hand side is the offset of the io space
under examination.   The centre block displays the hexadecimal
representation of each byte, and on the right hand side is their ascii
representation.   If a byte is not representable in ascii, then the
byte will be displayed as a dot.

By default the @command{dump} command reads from the currently
selected IO space.  However, it is possible to specify an explicit IO
space using the @code{ios} option:

@example
(poke) var myfile = open ("/path/to/file")
(poke) dump :ios myfile
@end example

@menu
* Information @command{dump} shows::         Changing what @command{dump} shows.
* Presentation options for @command{dump}::  Changing how @command{dump} shows it.
@end menu

@node Information @command{dump} shows
@subsection Information @command{dump} shows

By default @command{dump} displays 128 bytes of memory starting at offset 0#B.
You can change the quantity and starting offset by using the @code{size}
and @code{from} arguments.  For example:

@example
(poke) dump :from 0x10#B :size 0x40#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000010: b6a2 a578 8d82 7b7f 2076 374c 3eab 7150  ...x..@{. v7L>.qP
00000020: 31df 8ecb 3d33 ee12 429b 2e13 670d 948e  1...=3..B...g...
00000030: 86f1 2228 ae07 d95c 9884 cf0a d1a8 072e  .."(...\........
00000040: f93c 5368 9617 6c96 3d61 7b92 9038 a93b  .<Sh..l.=a@{..8.;
@end example

@noindent
Note that both the @code{size} and @code{from} arguments are offsets.
As such, both must be specified using @code{#} and an appropriate unit.
(@pxref{Offset Literals}).

The other arguments change the appearance of the dump.
If the @code{ruler} argument is zero, then the ruler will be omitted:

@example
(poke) dump :ruler 0 :size 0x40#B
00000000: b1fd 1608 2346 759c 46a6 aa94 6fcd 846a  ....#Fu.F...o..j
00000010: e39f 473f 3247 415f 174d a32b ed89 a435  ..G?2GA_.M.+...5
00000020: d2c6 2c52 bc82 e0a7 e767 31ea 84de 41e5  ..,R.....g1...A.
00000030: 2add 2869 e9c2 226b e222 8c74 4b94 af24  *.(i.."k.".tK..$
@end example

To omit the ascii
representation of the memory, call @command{dump} with the @code{ascii}
argument set to zero:

@example
(poke) dump :ruler 0 :size 0x40#B :ascii 0
00000000: 4393 85e7 0b0c 3921 5a26 39ec 2f5f 5f15
00000010: cc46 e6f3 d50f 6ae6 8988 d50e f8c4 d1c6
00000020: 5a2f 7c3e 490b 18d8 d867 4b6f 2549 1f6c
00000030: 34a9 a0d7 24d2 e9ac 9240 8247 10cb 4ba1
@end example

@node  Presentation options for @command{dump}
@subsection Presentation options for @command{dump}

By default, the hexadecimal display shows two bytes grouped together,
and then a space.   You can alter this behaviour using the @code{group_by}
parameter.

@example
(poke) dump :ascii 0 :size 0x40#B :group_by 4#B
00000000: 68f19a63 df2a8886 c466631c a7fdd5c7  h..c.*...fc.....
00000010: 3075746a 0adb03ca f5b1ff14 6166fa07  0utj........af..
00000020: 0dd3cfbd 8eff46a2 4152a81d 471beddf  ......F.AR..G...
00000030: a0501cae 8bfcec6f 7a4f5701 45ba9fc3  .P.....ozOW.E...
@end example

Another parameter is the @code{cluster_by} argument.
By setting @code{cluster_by} to @var{n}, this
causes @command{dump} to display an additional space after the @var{n}th
group has been displayed, and also in the corresponding position in
the ascii display:

@example
(poke) dump :size 0x40#B :group_by 2#B :cluster_by 4
76543210  0011 2233 4455 6677  8899 aabb ccdd eeff  01234567 89ABCDEF
00000000: 91b8 540d d4dc 49ae  3320 ba7d efd1 16ab  ..T...I. 3 .@}....
00000010: b1a8 5ea0 5846 8bea  f741 3f80 42bc 201f  ..^.XF.. .A?.B. .
00000020: 6e5e fa50 23fb f16a  d380 be8c fc98 d195  n^.P#..j ........
00000030: 7bbf fa3e 3fc2 43a4  2a1e 9763 2bd6 5d24  @{..>?.C. *..c+.]$
@end example

Another variable that the user can customize is
@code{pk_dump_nonprintable_char}.  It defaults to @code{'.'}, and
contains the character to use in the ASCII dump to represent a
non-printable character.

Not all bytes from an IO space can be read, depending on the kind of
IO device.  When @code{dump} cannot read a byte it will print
@code{pk_dump_unknown_byte} instead, which is a string.

If the variable @code{pk_dump_addr_hyperlinks} is 1 then the
@command{dump} command will emit hyperlinks in the printed bytes.
Clicking on a byte value will result on the address of that byte to be
inserted at the prompt.

If you have a personal preference on how memory dumps should appear,
you can set the relevant @code{pk_dump_*} variables.   These
global variables serve as the defaults for @command{dump}, so this way, you
will not need to explicitly pass them when you call the function.

@node copy
@section @command{copy}
@cindex @command{copy}

The command @command{copy} allows to copy regions of data inside an IO
space, or between different IO spaces.

This command has the following synopsis.

@example
copy [:from @var{offset}] [:to @var{offset}] [:size @var{offset}]
     [:from_ios @var{ios}] [:to_ios @var{ios}]
@end example

@noindent
All arguments are optional.  When invoked with no arguments,
@command{copy} does nothing.

The arguments @code{from} and @code{size} determine the region to
copy.  By default, this region is taken from the current IO space, but
this can be overwritten using the optional @code{from_ios} argument.

The argument @code{to} tells @command{copy} where to copy the stuff.
It defaults to @code{from}.  Again, this offset is applied to the
current IO space by default, but this can be overwritten using the
@code{to_ios} argument.

Note that it is allowed for the source and destination ranges to
overlap.

@node save
@section @command{save}
@cindex @command{save}

Use the @command{save} command in order to write a region from an IO
space into a file in your file system.

This command has the following synopsis.

@example
save [:ios @var{ios}] [:from @var{offset}] [:size @var{offset}] [:file @var{string}]
     [:append @var{bool}] [:verbose @var{bool}]
@end example

@noindent
All arguments are optional.  When invoked with no arguments,
@command{save} does nothing.

The arguments @code{from} and @code{size} are used to determine the
region of the IO space to save.  This is how you would extract and
save the contents of an ELF section to a file @file{out.text}:

@example
(poke) var s = elf_section_by_name (Elf64_Ehdr @@ 0#B, ".text")
(poke) save :file "out.text" :from s.sh_offset :size s.sh_size
@end example

@noindent
Both @code{from} and @code{size} are arbitrary offsets.  You should
keep in mind however that files are byte oriented.  Therefore saving,
say, nine bits to a file will actually write two bytes.

By default @command{save} will extract the data from the @dfn{current
IO space}.  However, it is possible to specify an alternative IOS by
using the @code{ios} argument.

By default @command{save} will truncate the output file, if it exists,
before starting writing.  If the argument @command{append} is
set as true, however, it will append to the existing contents of the
file.  In this case, the file should exist.

@node sdiff
@section @command{sdiff}
@cindex @command{sdiff}

The @command{sdiff} command provides structured byte differences
between two mapped values.

This command has the following synopsis:

@example
sdiff :a @var{any} :b @var{any} [:group_by @var{offset}] \
      [:values @var{bool}]
@end example

@noindent
Where @var{:a} and @var{:b} are two mapped values of the same type.

By default, the bytes are shown in the diff separated by white
characters.  You can alter this behaviour using the @code{group_by}
parameter.

If @var{values} is true, then include field interpreted values in the
diff output.

If the two values are not mapped, this command raises @code{E_no_map}.
If the two values are not of the same type, the output will probably
not be meaningful.

@node extract
@section @command{extract}
@cindex @command{extract}

Use the @command{extract} command in order to create a temporary
memory IO space with the contents of a mapped value.

This command has the following synopsis.

@example
extract :val @var{val} [:to @var{ios_name}]
@end example

@noindent
Where @code{:val} is a mapped value, and @var{:to} is the name of the
memory IOS to create (or use).  The contents of @var{value} are copied
to the beginning of the memory IOS @code{*@var{ios_name}*}.

If no @var{:to} argument is specified, or if it is the empty string,
then the @command{extract} command will generate a suitable unique
name and use it.

@node scrabble
@section @command{scrabble}
@cindex @command{scrabble}

The @command{scrabble} command rearranges portions of an IO space
based on a transformation defined by a pair of patterns.

This command has the following synopsis:

@example
scrabble [:from @var{offset}] [:size @var{offset}]
         [:from_pattern @var{string}] [:to_pattern @var{string}]
         [:ent_size @var{offset}]
         [:from_ios @var{ios}] [:to_ios @var{ios}]
@end example

@noindent
The optional arguments are documented below.

@table @code
@item :from @var{offset}
Beginning of the range in the origin IO space from where to start
rearranging units.  Defaults to @code{0#B}.
@item :size @var{offset}
Size of the range to scrabble in the origin IO space.  Defaults to
@code{0#B}.
@item :from_pattern @var{string}
String specifying a sequence of units.  Each character is an unit.
Defaults to @code{""}.
@item :to_pattern @var{string}
String specifying the scrabbled units.  Each character is an unit.  If
a character that doesn't appear in the @code{from_pattern} is found in
this string, it is ignored.   Defaults to @code{from_pattern}.
@item :ent_size @var{offset}
Size of the entities to scrabble around.  Defaults to @code{1#B}.
@item :from_ios @var{ios}
IO space from where to obtain the data to scrabble.  Defaults to the
current IO space.
@item :to_ios @var{ios}
The IO space to where write the scrabbled data.  Defaults to
@code{from_ios}.
@end table

@c XXX add usage examples for scrabble.

@node The Poke Language
@chapter The Poke Language

@menu
* Integers::			Whole numbers.
* Offsets::			Memory sizes and offsets.
* Strings::			NULL-terminated strings.
* Arrays::			Homogeneous collections.
* Structs::			Heterogeneous collections.
* Types::			Declaring types.
* Assignments::			Changing the value of variables.
* Compound Statements::		Sequences of statements.
* Conditionals::		Conditional statements and expressions.
* Loops::			Statements to iterate on conditions.
* Expression Statements::	Using expressions for their side-effects.
* Functions::			Procedural abstraction.
* Assembler::                   Assembler instructions with Poke operands.
* Endianness::			Byte ordering.
* Mapping::			Accessing IO spaces.
* Exception Handling::		Dealing with exceptional conditions.
* Terminal::	                Dealing with the terminal.
* Printing::			Output in Poke programs.
* Comments::			Documenting Poke programs.
* Modules::			Loading pickles from Poke programs.
* System::                      Accessing the system from within Poke.
* VM::                          Accessing the PVM in Poke programs.
* Debugging::                   Debugging Poke programs.
@end menu

@node Integers
@section Integers
@cindex integers

Most of the values manipulated in Poke programs are whole numbers,
also typically known as @dfn{integers} in computing parlance.  This is
because integers are pervasive in binary formats, often featuring
unusual characteristics in terms of size and/or alignment.  Single
bits denoting flags or packed small integers are good examples of
this.  In order to ease the manipulation of such entities, and unlike
most programming languages, Poke provides integer types of any number
of bits and a rich set of accompanying operators.

@menu
* Integer Literals::		Writing integers in different bases.
* Characters::			Another way to write small integers.
* Booleans::			Integers denoting truth values.
* Integer Types::		int<N> and uint<N>.
* Casting Integers::		Converting integers.
* Relational Operators::	Comparing integers.
* Arithmetic Operators::	Operating with integers.
* Bitwise Operators::		Getting and setting integer bits.
* Boolean Operators::		Equality and inequality.
* Integer Attributes::		Accessing properties of integer values.
@end menu

@node Integer Literals
@subsection Integer Literals
@cindex integer literals
Integers literals can be expressed in several numeration bases.

@cindex decimal
@emph{Decimal numbers} use the usual syntax @code{[1-9][0-9]*}.  For
example, @code{2345}.

@cindex octal
@emph{Octal numbers} are expressed using a prefix @code{0o} (or
@code{0O}) followed by one or more digits in the range @code{[0-7]}.
Examples are @code{0o0}, @code{0o100} and @code{0o777}.

@cindex hexadecimal
@emph{Hexadecimal numbers} are expressed using a prefix @code{0x} (or
@code{0X}) followed by one or more hexadecimal digits in the range
@code{[0-f]}.  Examples are @code{0x0} and @code{0xfe00ffff}.  Note
that both the @code{x} in the prefix and the letters in the
hexadecimal number are case insensitive.  Thus, @code{0XdeadBEEF} is a
valid (but ugly as hell) literal.

@cindex binary
@emph{Binary numbers} are expressed using a prefix @code{0b} (or
@code{0B}) followed by one or more binary digits in the range
@code{[0-1]}.  Examples of binary literals are @code{0b0} and
@code{0B010}.

Negative numbers, of any numeration base, are constructed using the
minus operator as explained below.  Therefore the minus symbol
@code{-} in negative numbers is not part of the literal themselves.

@subsubsection The digits separator @code{_}
@cindex digits separator
@cindex separator, digits separator
The character @code{_} can appear anywhere in a numeric literal
except as the first character.  It is ignored, and its purpose is to
make it easier for programmers to read them:

@example
0xf000_0000_0000_0000
0b0000_0001_0000_0001
@end example

@subsubsection Types of integer literals
@cindex integer literals
@cindex size of variables
@cindex bitsize
The type of a numeric literal is the smallest signed integer capable
of holding it, starting with 32 bits, in steps of powers of two and up
to 64 bits.@footnote{Rationale: the width of a C ``int'' is 32 bits in
most currently used architectures, and binary data formats are usually
modelled after C.}

So, for example, the value @code{2} has type @code{int<32>}, but the
value @code{0xffff_ffff} has type @code{int<64>}, because it is out of
the range of signed 32-bit numbers.

A set of suffixes can be used to construct integer literals of certain
types explicitly.  @code{L} or @code{l} is for 64-bit integers.
@code{H} or @code{h} is for 16-bit integers (also known as
@dfn{halves}), @code{B} or @code{b} is for 8-bit integers (also known
as @dfn{bytes}) and @code{n} or @code{N} is for 4-bit integers (also
known as @dfn{nibbles}).

Thus, @code{10L} is a 64-bit integer with value
@code{0x0000_0000_0000_000A}, @code{10H} is a 16-bit integer with
value @code{0x000A} and @code{10b} is a 8-bit integer with value
@code{0x0A}.

@cindex signedness
Similarly, the signed or unsigned attribute of an integer can be
explicitly specified using the suffix @code{u} or @code{U} (the
default are signed types).  For example @code{0xffff_ffffU} has type
@code{uint<32>} and @code{0ub} has type @code{uint<8>}.  It is possible
to combine width-indicating suffixes with signedness suffixes:
@code{10UL} denotes the same literal as @code{10LU}.

The above rules guarantee that it is always possible to determine the
width and signedness of an integer constant just by looking at it,
with no ambiguity.

@node Characters
@subsection Characters
@cindex characters
8-bit unsigned integers can use an alternative literal notation that
is useful when working with @emph{ASCII character codes}.  Printable
character codes can be denoted with @code{'c'}.

@cindex escape sequence
Non-printable characters can be expressed using escape-sequences.  The
allowed sequences are:

@table @code
@item \n
New-line character (ASCII 012).
@item \t
Tab character (ASCII 011).
@item \\
The backslash character.
@item \[0-9][0-9]?[0-9]?
Character whose ASCII code is the specified number, in octal.
@end table

Examples:

@example
'o'
'\n'
'\t'
'\\'
'\0'
@end example

The type of a character literal is always @code{char}, aka
@code{uint<8>}.

@node Booleans
@subsection Booleans
@cindex boolean values
Like in C, truth values in Poke are encoded using integers.  Zero
(@code{0}) denotes the logical value ``false'', and any integer other
than zero denotes the logical value ``true''.

@node Integer Types
@subsection Integer Types

Most general-purpose programming languages provide a small set of
integer types, each featuring a range corresponding to strategic
storage sizes: basically, signed and unsigned variants of 8, 16, 32,
64 bits.  As we have seen in the previous sections, suffixes like
@code{H} or @code{L} are used in Poke for that purpose.

However, in conventional programming languages when integers having an
``odd'' width (like 13 bits, for example) get into play for whatever
reason, the programmer is required to use the integer arithmetic
operators (and sometimes bit-wise operators) herself, in a clever way,
in order to achieve the desired results.

Poke, on the contrary, provides a rich set of integer types featuring
different widths, in both signed and unsigned variants.  The language
operators are aware of these types, and will do the right thing when
operating on integer values having different widths.

Unsigned integer types are specified using the type constructor
@code{uint<@var{n}>}, where @var{n} is the number of bits.  @var{n}
should be an integer literal in the range @code{[1,64]}.  Examples:

@example
uint<1>
uint<7>
uint<64>
@end example

Similarly, signed integer types are created using the type constructor
@code{int<@var{n}>}, where @var{n} is the number of bits.  @var{n}
should be an integer literal in the range @code{[1,64]}.  Examples:

@example
int<1>
int<8>
int<64>
@end example

Note that expressions are not allowed in the type integral constructor
parameters.  Not even constant expressions.  Thus, things like
@code{int<foo>} and @code{uint<2+3>} are not allowed.

@node Casting Integers
@subsection Casting Integers
@cindex casts
The right-associative unary operator cast @code{as} can be used to
derive a new integer value having a different type from an existing
value.

For example, this is how we would create a signed 12-bit integer value
holding the value 666:

@example
(poke) 666 as int<12>
(int<12>) 666
@end example

Note that the @code{666} literal is originally a 32-bit signed
integer.  The cast performs the conversion.

Casts between integer types are always allowed, and never fail.  If
the new type is narrower than the existing type, truncation may be
performed to accommodate the value in its new type.  For example, the
expression @code{0x8765_4321 as uint<16>} evaluates to @code{0x4321}.
If the new type is wider than the existing type, either zero-extension
or sign-extension is performed depending on the signedness of the
operand.

@cindex signedness
The semantics of the sign-extension operation depends on the
signedness of the value being converted, and on the currently
selected encoding for negative numbers.

When using two's complement encoding, converting a signed value will
always sign-extend regardless of the signedness of the target type.
Thus:

@example
(poke) -2H as uint<32>
0xfffffffeU
(poke) -2H as int<32>
0xfffffffe
@end example

Likewise, converting an unsigned value will always zero-extend
regardless of the signedness of the target type. Thus:

@example
(poke) 0xffffUH as uint<32>
0xffffU
(poke) 0xffffUH as int<32>
0xffff
@end example

@node Relational Operators
@subsection Relational Operators

@cindex comparison
The following binary relational operators are supported on integer
values, in descending precedence order:

@itemize
@item Equality @code{==} and inequality @code{!=}.
@item Less than @code{<} and less or equal than @code{<=}.
@item Greater than @code{>} and greater or equal than @code{>=}.
@end itemize

When applied to integer and character values, these operators
implement an arithmetic ordering.

These operators resolve in boolean values encoded as 32-bit integers:
@code{0} meaning false and @code{1} meaning true.

@node Arithmetic Operators
@subsection Arithmetic Operators
@cindex arithmetic
The following left-associative binary arithmetic operators are
supported, in descending precedence order:

@itemize
@item Exponentiation @code{**}, multiplication @code{*}, integer division @code{/}, integer
ceil-division @code{/^} and modulus @code{%}.
@item Addition @code{+} and subtraction @code{-}.
@end itemize

@cindex signedness
@cindex sign promotion
In all the binary arithmetic operations automatic promotions
(coercions) are performed in the operands as needed.  The rules are:

@itemize @bullet
@item If one of the operands is unsigned and the other operand is
signed, the second is converted to an unsigned value.
@item If the size in bits of one of the operands is bigger than the
size of the other operand, the second is converted to the same number
of bits.
@end itemize

The following right-associative unary arithmetic operators are
supported:

@itemize
@item Unary minus @code{-} and unary plus @code{+}.
@cindex minus
@end itemize

Finally, pre-increment, pre-decrement, post-increment and
post-decrement operators @code{++} and @code{--} are supported.

@node Bitwise Operators
@subsection Bitwise Operators
@cindex bitwise operators
The following left-associative bitwise binary operators are supported,
in descending precedence order:

@itemize
@item Bitwise shift left @code{<<.} and bitwise shift right @code{.>>}.
@cindex shifting
@item Bitwise AND @code{&}.
@cindex AND
@item Bitwise exclusive OR @code{^}.
@cindex exclusive OR
@item Bitwise inclusive OR @code{|}.
@cindex inclusive OR
@cindex OR
@item Bitwise concatenation @code{:::}.
@cindex concatenation, bitwise
@end itemize

Both @code{<<.} and @code{.>>} operators perform logical shifting.
Unlike in many other programming languages, arithmetic right-shifting
operators are not provided.  This means that right shifting always
inserts zeroes at the most-significant side of the value operand,
whereas left shifting always inserts zeroes at the least-significant
side of the value operand.

Left shifting by a number of bits equal or bigger than the size of the
value operand is an error, and will trigger either a compile-time
error or a run-time @code{E_out_of_bounds} exception.  This does not
apply to right shifting.

Bitwise concatenation works with any integral type, of any bit length.

The following right-associative unary bitwise operators are supported:

@itemize
@item Bitwise complement @code{~}.
@cindex complement
@end itemize

@node Boolean Operators
@subsection Boolean Operators
@cindex boolean operators
The following left-associative, short-circuited binary logical
operators are supported, in descending precedence order:

@itemize
@item Logical implication: @code{=>}.
@item Logical AND: @code{&&}.
@item Inclusive OR: @code{||}.
@end itemize

The following right-associative unary logical operators are supported:

@itemize
@item Logical negation @code{!}.
@cindex negation
@end itemize

@node Integer Attributes
@subsection Integer Attributes
@cindex attributes, integer attributes

The following attributes are defined for integer values.

@table @code
@item size
Gives an offset with the storage occupied by the string.  This
includes the terminating null.  Examples:

@example
(poke) 10'size
0x20UL#b
(poke) 10N'size
0x4UL#b
(poke) (10 as int<1>)'size
0x1UL#b
@end example

@item signed
@cindex signedness
Gives 1 if the value is a signed integer, 0 otherwise.  Examples:

@example
(poke) 10'signed
1
(poke) 10UL'signed
0
@end example
@item mapped
Always 0 for integers.  (@pxref{Mapping}).
@item length
Always @code{1UL} for integers.
@end table


@node Offsets
@section Offsets
@cindex offset
@cindex united values

Poke uses united values to handle offsets and data sizes.  This is a
very central concept in poke.

@menu
* Offset Literals::		Denoting offsets in Poke.
* Offset Units::		Pears and potatoes.
* Offset Types::		offset<@dots{}>.
* Casting Offsets::		Converting offsets.
* Offset Operations::		Operating with offsets.
* Offset Attributes::           Accessing properties of offset values.
@end menu

@node Offset Literals
@subsection Offset Literals

Poke provides a convenient syntax to provide united values, which are
called @dfn{offsets} (because in a binary editor you mostly use them
to denote offsets in the file you are editing):

@example
12#B
7#b
1024#KB
@end example

@cindex magnitude
@cindex kilobytes
The offsets above denote twelve bytes, seven bits and one thousand
twenty four kilobytes, respectively.  The unit can be separated from
the magnitude by blank characters, so you can write the following
instead if you are so inclined:

@example
12 #B
7 #b
(1024 * 1024) #Kb
@end example

Note how the magnitude part of an offset doesn't need to be constant.
If the variable @code{a} contains an integer, this is how you would
denote ``a bytes'':

@example
a#B
@end example

In the offset syntax units are specified as @code{#@var{unit}}, where
@var{unit} is the specification of an unit.  See the next section for
details.

@node Offset Units
@subsection Offset Units

There are several ways to express the unit of an offset, which is
always interpreted as a multiple of the basic unit, which is the bit
(one bit).

@subsubsection Named Units

The first way is to refer to an unit by name.  For example, @code{2#B}
for two bytes.  Units are defined using the @code{unit}
construction:

@example
unit @var{name} = @var{constant_expression} [, @var{name} = @var{constant_expression}]...;
@end example

@noindent
where @var{name} is the name of the unit, and
@var{constant_expression} is a constant expression that should
evaluate to an integral value.  The resulting value is always coerced
into an unsigned 64-bit integer.

Note that unit names live in a different namespace than variables and
types.  However, when a given name is both a type name and an unit
name in an unit context, the named unit takes precedence:

@example
(poke) type xx = int
(poke) unit xx = 2
(poke) 1#xx
1#2
@end example

Many useful units are defined in the standard library.  @xref{Standard
Units}.

@subsubsection Arbitrary Units

It is also possible to express units in multiples of the base unit,
which is the bit.  Using this syntax, it becomes possible to express
offsets in any arbitrary unit, as disparate as it may seem:

@example
17#3
0#12
8#1
@end example

That's it: 17 units of 3 bits each, zero units of 12 bits each,
and eight units of 1 bit each.  Note that the unit should be greater
than 0.

@subsubsection Types as Units

But then, why stop there?  Poking is all about defining data
structures and operating on them@dots{} so why not use these structures
as units as well?  Consider the following struct:

@example
type Packet = struct @{ int i; long j; @};
@end example

The size of a @code{Packet} is known at compile time (which is not
generally true for Poke structs).  Wouldn't it be nice to use it as a
unit in offsets?  Sure it is:

@example
23#Packet
@end example

The above is the size occupied by 23 packets.  Any type whose size is
known at compile time can be specified as an offset unit.

@cindex ELF
Expressing offsets as united values also relieves the programmer from
doing many explicit unit conversions: poke can do them for you.
Consider for example an ELF section header.  One of its fields is the
size of the described section, in bytes:

@example
type Elf64_Shdr =
  struct
  @{
   @dots{}
   offset<Elf64_Xword,B> sh_size;
   @dots{}
  @};
@end example

If a given section is to contain, say, relocations with addends,
we can set its size doing something like this:

@example
shdr.sh_size = 10#Elf64_Rela;
@end example

@noindent
Instead of doing the conversion to bytes explicitly.

@cindex magnitude
If the magnitude of an offset is 1 then it is allowed to omit it
entirely.  To denote one kilobyte, for example, we can write
@code{#KB}.

@node Offset Types
@subsection Offset Types

Offset types are denoted as @code{offset<@var{base_type},@var{unit}>},
where @var{base_type} is an integer type and @var{unit} the
specification of an unit.

The offset base type is the type of the magnitude part of the united
value.  It can be any integer type, signed or unsigned, of any size.

The unit specification should be one of the unit identifiers that are
allowed in offset literals (see above), a constant positive integer or
the name of a Poke type whose size is known at compile time.

@cindex kilobits
Let's see some examples.  A signed 32-bit offset expressed in bytes
has type @code{offset<int<32>,B>}.  An unsigned 12-bit offset
expressed in kilobits has type @code{offset<uint<12>,Kb>}.  The latter
type can also be written using an explicit integer unit like in
@code{offset<uint<12>,1024>}.  Finally, a signed 64-bit offset in units
of ``packets'', where a packet is denoted with a Poke type
@code{Packet} has type @code{offset<uint<64>,Packet>}.

@node Casting Offsets
@subsection Casting Offsets
@cindex casts
The right-associative unary operator cast @code{as} can be used to
derive a new offset value having a different type from an existing
value.

For example, this is how we would create a signed 12-bit offset in
units bytes:

@example
(poke) 1024#b as offset<int<12>,B>
(int<12>) 128#B
@end example

@cindex magnitude
The same rules governing conversion of integers apply for the
magnitude part.  Depending on the unit, there can be truncation, like
in:

@example
(poke) 9#b as offset<int,B>
1#B
@end example

@node Offset Operations
@subsection Offset Operations
@cindex offset algebra
Poke supports a little algebra for offsets.

@subsubsection Addition and subtraction
@cindex addition
@cindex subtraction
The addition or subtraction of two offsets results in another offset.
Examples:

@example
(poke) 1#B + 1#b
9#b
(poke) 2#KB - 1024#B
1024#B
@end example

The unit of the result is the greatest common divisor of the units of
the operands.

The operators @code{++} and @code{--}, in their prefix and suffix
versions, can be applied to offsets as well.  The step used in the
increment/decrement is one unit.

@subsubsection Multiplication by a scalar
@cindex multiplication
Multiplying an offset by a magnitude gives you another offset.
Examples:

@example
(poke) 8#b * 2
16#b
(poke) 16#MB * 0
0#MB
@end example

The unit of the result is the same as the unit of the offset
operand.

Note that multiplying two offsets is not supported.  This makes sense,
since computer memory is linear, and therefore it wouldn't make any
sense to have units like @code{#B@sup{2}}.

@subsubsection Division
@cindex division
Dividing two offsets gives you a magnitude.  Examples:

@example
(poke) 16#b / 1#B
2
(poke) 1024#MB / 512#Mb
16
@end example

Dividing offsets is the Pokish way of converting memory magnitudes
between different units: just use units like you do when doing physics
or working with units in other contexts.

@cindex kilobits
For example, using the syntactic trick of omitting the magnitude (in
which case it is assumed to be 1) it is very natural to write
something like the following to convert from kilobits to bytes:

@example
(poke) 24 #Kb/#B
3072
@end example

There is also a ceil-division operator for offsets, with the same
semantics as the ceil-division for integers:

@example
(poke) 10#B /^ 3#B
4
@end example

@subsubsection Division by an integer

Dividing an offset by an integer gives you an offset.  Example:

@example
(poke) 8#B / 2
4#B
@end example

@subsubsection Modulus
@cindex modulus
The modulus of two offsets gives you another offset with the expected
semantics.  Examples:

@example
(poke) 9#b % 1#B
1#b
(poke) 1#B % 9#b
8#b
@end example

The unit of the result is the greatest common divisor of the units of
the operands.

@node Offset Attributes
@subsection Offset Attributes
@cindex attributes, offset
The following attributes are defined for offset values.

@table @code
@item size
Gives an offset with the storage occupied by the offset.  Examples:

@example
(poke) 10#B'size
0x20UL#b
(poke) 10N#B'size
0x4UL#b
@end example
@item magnitude
Gives the magnitude part of the offset.  Examples:

@example
(poke) 10#B'magnitude
10
(poke) 2H#b'magnitude
2H
@end example
@item unit
Gives a number with the unit of the offset, expressed in bits.
Examples:

@example
(poke) 10#B'unit
8UL
(poke) 2H#b'unit
1UL
@end example
@item mapped
Always 0 for offsets.  (@pxref{Mapping}).
@item length
Always @code{1UL} for integers.
@end table


@node Strings
@section Strings
@cindex strings

Poke supports a notion of @dfn{strings} which is very similar to the C
programming language: a string value is a sequence of characters that
is terminated by the so-called @dfn{null character}.

The standard library provides functions which process strings.
@xref{String Functions}.

@menu
* String Literals::		Writing string values.
* String Types::		string.
* String Indexing::		Accessing the characters of a string.
* String Concatenation::        Building new strings.
* String Attributes::           Accessing properties of string values.
* String Formatting::           Composing strings from values.
@end menu

@node String Literals
@subsection String Literals

NULL-terminated sequences of ASCII codes can be denoted using the
following syntax:

@example
"foo"
@end example

Poke string values are very similar to C strings.  They comprise a
sequence of 8-bit character codes, terminated by the value @code{0UB}.

The following escape sequences are supported inside string literals:

@table @code
@item \n
Denotes a new line character.
@item \t
Denotes an horizontal tab.
@item \\
Denotes a backlash @code{\} character.
@item \"
Denotes a double-quote @code{"} character.
@item \@var{num}
@var{num} is a number in the range 1 to 255 (inclusive) in base 8.
@item \x@var{num}
@var{num} is a number in the range 1 to 255 (inclusive) in base 16.
@end table

Strings cannot contain @code{0UB} character, and inserting one using
escape sequences (@code{\0} or @code{\x0}) leads to compilation error.

@node String Types
@subsection String Types

Every string value in Poke is of type @code{string}.

@node String Indexing
@subsection String Indexing
@cindex indexing, into strings
Poke supports accessing the characters in a string using the array
indexing notation.  The indexes are in the @code{[0,@var{n}]} range,
where @var{n} is the length of the string minus one.  Note the length
doesn't include the null character, @i{i.e.} it is not possible to access
the terminating null.  Examples:

@example
(poke) "foo"[0]
0x66UB
(poke) "foo"[1]
0x6fUB
@end example

@cindex exceptions
If the passed index is less than zero or it is too big, an
@code{E_out_of_bounds} exception is raised:

@example
(poke) "foo"[-1]
unhandled out of bounds exception
(poke) "foo"[3]
unhandled out of bounds exception
@end example

@noindent
Poke also provides a way to refer to a character in a string by the
offset the character occupies in the string.  This is done by using as
index an expression that evaluates to an offset value.  This value is
then implicitly promoted to have an unsigned 64-bit magnitude and unit
bits whenever needed.  If no character starts @emph{exactly} at the
provided offset then an @code{E_out_of_bounds} exception is raised.

@node String Concatenation
@subsection String Concatenation
@cindex concatenation, strings
Strings can be concatenated using the @code{+} operator.  This works
like this:

@example
(poke) "foo" + "bar"
"foobar"
@end example

Note how the null character terminating the first string is removed.
Therefore, the length of the concatenation of two given strings
of lengths @code{N} and @code{M} is always @code{N+M-1}.

Concatenation and indexing are useful together for building strings.
A string can be created empty, and additional characters added to it
by means of concatenation:

@example
(poke) var bytes = "";
(poke) bytes = bytes + 'x' as string;
@end example

Then, we can retrieve characters from the string we built using
indexing:

@example
(poke) bytes[0]
0x78UB
@end example

Additionally, the @code{*} operator allows to ``multiply'' a string by
concatenating it with itself a given number of times.  This works like
this:

@example
(poke) "foo" * 3
"foofoofoo"
(poke) "foo" * 0
""
@end example

This is useful for building strings whose length is not known at
compile time.  For example:

@example
fun make_empty_string = (int length) string:
@{
   return " " * length;
@}
@end example

@node String Attributes
@subsection String Attributes
@cindex attributes, string attributes
The following attributes are defined for string values.

@table @code
@item length
Gives the number of characters composing the string, not counting the
terminating null.  Examples:

@example
(poke) "foo"'length
3UL
(poke) ""'length
0UL
@end example

@item size
Gives an offset with the storage occupied by the string.  This
includes the terminating null.  Examples:

@example
(poke) "foo"'size
32UL#b
(poke) ""'size
8UL#b
@end example
@item mapped
Always 0 for strings.  (@pxref{Mapping}).
@end table

@node String Formatting
@subsection String Formatting
@cindex strings, formatting

Poke provides a built-in function @code{format} that can be used in
order to format, or build, a string value that is derived from a set
of zero or more Poke values:

@example
format (@var{fmt}[, @var{value}@dots{}])
@end example

@noindent
Where @var{fmt} is a format string that interpreted verbatim but for
predefined @dfn{format tags} which start with @code{%}.  These format
tags are interpreted especially.  @xref{printf}.

Example:

@example
var s = format ("Age is %i32d.", 66);
@end example

@noindent
The value of the variable @code{s} is @code{"Age is 66."}.

@node Arrays
@section Arrays
@cindex arrays

Arrays are homogeneous collections of values.

@menu
* Array Literals::		Writing array values.
* Array Types::			Bounded and unbounded arrays.
* Casting Arrays::              Converting arrays from one type to another.
* Array Constructors::		Constructing array values.
* Array Comparison::		Comparing array values.
* Array Indexing::		Accessing values stored in arrays.
* Array Trimming::		Working with array pieces.
* Array Elements::		Checking for array elements.
* Array Concatenation::		Concatenating the elements of arrays.
* Array Attributes::		Accessing properties of array values.
* Arrays as Stacks::            Using arrays as stacks.
@end menu

@node Array Literals
@subsection Array Literals

Array literals are constructed using the following syntax:

@example
[@var{exp},@var{exp}@dots{}]
@end example

@noindent
where @var{exp} is an arbitrary expression.

For example, @code{[1,2,3]} constructs an array of three signed 32-bit
integers.  Likewise, @code{['a','b','c']} constructs an array of three
unsigned 8-bit integers (characters).  For convenience, a trailing
comma is accepted but not required.

The type of the array literal is inferred from the type of its
elements.  Accordingly, all the elements in an array literal must be
of the same type.  Examples of invalid array literals, that will raise
a compilation-time error if evaluated, are:

@example
[1,2u,3]
[1,0xffff_ffff,3]
['a',"b",'c']
@end example

Array literals must contain at least one element.  Accordingly,
@code{[]} is not a valid array literal.

@cindex matrices
This is how a @code{3x3} matrix could be constructed using an array of
arrays:

@example
[[1,2,3],[4,5,6],[7,8,9],]
@end example

It is possible to refer to specific elements when constructing array
literals.  For example, @code{[1,2,3,.[3] = 4]} denotes the same
array as @code{[1,2,3,4]}.

This allows creating arrays without having to specify all its
elements; every unspecified element takes the value of the first
specified element to its right.  For example, @code{[.[2] = 2]}
denotes the same array as @code{[2,2,2]}.

Note that an array element can be referenced more than once.  When
that happens, the final value of the element is the last specified.
For example, @code{[1,2,3,.[1]=10]} denotes the array @code{[1,10,3]}.

@node Array Types
@subsection Array Types

There are three different kind of array types in Poke.

@dfn{Unbounded} arrays have no explicit boundaries.  Examples are
@code{int[]} or @code{Elf64_Shdr[]}.  Arrays can be @dfn{bounded by
number of elements} specifying a Poke expression that evaluates to an
integer value.  For example, @code{int[2]}.  Finally, arrays can be
@dfn{bounded by size} specifying a Poke expression that evaluates to
an offset value.  For example, @code{int[8#B]}.

@subsubsection Writing unbounded array literals
@cindex unbounded arrays
The type of an array literal is always bounded by number of elements.
For example, the type of @code{[1,2,3]} is @code{int[3]}.  If what we
want is an unbounded array literal we can obtain it with a case like
@code{[1,2,3] as int[]}.

@subsubsection Array boundaries and closures
@cindex closures
Poke arrays are rather peculiar.  One of their seemingly bizarre
characteristics is the fact that the expressions calculating their
boundaries (when they are bounded) evaluate in their own lexical
environment, which is captured.  In other words: the expressions
denoting the boundaries of Poke arrays conform closures.  Also, the
way they evaluate may be surprising.  This is no capricious.

When an array type is bounded, be it by number of elements or by size,
the expression indicating the boundary doesn't need to be constant and
it can involve variables.  For example, consider the following type
definition:

@example
var N = 2;
type List = int[N*2];
@end example

@noindent
Let's map a @code{List} at some offset:

@example
(poke) List @@ 0#B
[0x746f6f72,0x303a783a,0x723a303a,0x3a746f6f]
@end example

As expected, we get an array of four integers.  Very good,
obviously the boundary expression @code{N*2} got evaluated
when defining the type @code{List}, and the result of the
evaluation was @code{4}, right?.  Typical semantics like in my
garden variety programming language@dots{}?  Right?!?

Well, not really.  Let's modify the value of @code{N} and map
a @code{List} again@dots{}

@example
(poke) N = 1
(poke) List @@ 0#B
[0x746f6f72,0x303a783a]
@end example


Yes, The boundary of the array type changed@dots{} on, this is Poke,
was you @strong{really} expecting something typical? ;)

What happens is that at type definition time the lexical environment
is captured and a closure is created.  The body of the closure is the
expression.  Every time the type is referred, the closure is
re-evaluated and a new value is computed.

Consequently, if the value of a variable referred in the expression
changes, like in our example, the type itself gets updated
auto-magically.  Very nice but, why is Poke designed like this?  Just
to impress the cat?  Nope.

In binary formats, and also in protocols, the size of some given data
is often defined in terms of some other data that should be decoded
first.  Consider for example the following definition of a
@code{Packet}:

@example
type Packet =
  struct
  @{
    byte size;
    byte[size] payload;
  @};
@end example

Each packet contains a 8-bit integer specifying the size of the
payload transported in the packet.  The payload, a sequence of
@code{size} bytes, follows.

In struct types like the above, the boundaries of arrays depend on
fields that have been decoded before and that exist, like variables,
in the lexical scope captured by the struct type definition (yes,
these are also closures, but that's for another article.)  This
absolutely depends on having the array types evaluate their bounding
expressions when the type is used, and not at type definition time.


@noindent
To show this property in action, let's play a bit:

@example
(poke) var data = byte[4] @@ 0#B
(poke) data[0] = 2
(poke) data[1] = 3
(poke) data[2] = 4
(poke) data[3] = 5
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
00000000: 0203 0405 0000 0000 0000 0000 0000 0000
00000010: 0000 0000 0000 0000 0000 0000 0000 0000
(poke) var p1 = Packet @@ 0#B
(poke) var p2 = Packet @@ 1#B
(poke) p1
Packet @{size=0x2UB,payload=[0x3UB,0x4UB]@}
(poke) p2
Packet @{size=0x3UB,payload=[0x4UB,0x5UB,0x0UB]@}
@end example

Now, let's change the data and see how the sizes of the payloads
are adjusted accordingly:

@example
(poke) data[0] = 1
(poke) data[1] = 0
(poke) p1
Packet @{size=0x1UB,payload=[0x0UB]@}
(poke) p2
Packet @{size=0x0UB,payload=[]@}
@end example

So, as we have seen, Poke's way of handling boundaries in array types
allows data structures to adjust to the particular data they contain,
so usual in binary formats.  This is an important feature, that gives
Poke part of its feel and magic.

@node Casting Arrays
@subsection Casting Arrays
@cindex casting, integration, integrate, arrays

There are two kinds of casts for array values:

@itemize @bullet
@item
Cast to array types that may have a different bounding.
@item
Cast of array of integers or integral structs to an integral type.
@end itemize

For example, we have seen that the type of an array literal is always
bounded by a constant number of elements.  Thus the type of
@code{[1,2,3]} is @code{int[3]}.

We could cast the value above to an unbounded array type by doing:

@example
[1,2,3] as int[]
@end example

@noindent
Or to an array bounded by size:

@example
[1,2,3] as int[12#B]
@end example

Of course trying to cast an array value to a type for which the
bounding restrictions will fail.  This will give you a compile-time
error:

@example
[1,2,3] as int[4]
@end example

@noindent
And this will result in a run-time error:

@example
(poke) [1,2,3] as int[13#B]
unhandled conversion error exception
@end example

Now let's see how Poke reinterprets integral arrays as integers:

@example
(poke) [0xdeUB,0xadUB] as uint16
0xdeadUH
(poke) [0xdeUB,0xadUB,0UB] as int
0xdead00
@end example

Trying to cast an array with a size larger than 64-bit will result
in an error:

@example
(poke) [1,2,3] as int
unhandled conversion error exception
arrays bigger than 64-bit cannot be casted to integral types
@end example

It also works for nested arrays and arrays of integral structs
(@xref{Integral Structs}):

@example
(poke) type Foo = struct uint @{ int16 hi; int16 lo; @};
(poke) [[Foo@{ hi=0xdeadH, lo=0xbeefH@}],[Foo@{@}]] as ulong
0xdeadbeef00000000UL
@end example

The way it works is like putting all array elements in an
@code{uint<64>} number and then casting that number to the
destination integral type.

@node Array Constructors
@subsection Array Constructors
@cindex constructing, arrays

Array literals can only specify arrays which a constant number of
elements.  They also require initial values for their elements, and it
is also not possible to construct an array literal denoting an empty
array of some given type.

Array constructors provide a suitable way to build such array values.
The syntax is:

@example
@var{array_type} ([@var{value}])
@end example

@noindent
Where @var{array_type} is an array type, or the name of an array type,
and @var{value} is an optional argument that specify the value to
which the elements of the new array are initialized.  If no value is
provided in the constructor then default values are calculated.

If the boundaries of the array can't be satisfied while constructing,
@code{E_constraint} is raised.

Examples:

@example
(poke) int[]()
[]
(poke) int[2]()
[0,0]
(poke) int[2](10)
[10,10]
(poke) offset<int,B>[2](16#b)
[2#B,2#B]
(poke) string[3]()
["","",""]
(poke) int[][3](int[2](666))
[[666,666],[666,666],[666,666]]
(poke) Packet[3]()
[Packet @{...@}, Packet @{...@}, Packet @{...@}]
@end example

Note that array constructors are implicitly involved when you
construct a struct or union value that contains a field whose type is
an array.  For example, consider the following type:

@example
type Packet =
  struct
  @{
    byte sz;
    byte[sz] data;
  @};
@end example

@noindent
When a packet is constructed using @code{Packet @{@}} the @code{data}
field of the struct is also constructed, implicitly using a
constructor for the array type @code{byte[sz]}.

@node Array Comparison
@subsection Array Comparison
@cindex comparing, arrays

The equality operator (@code{==}) and the inequality operator
(@code{!=}) can be applied to arrays.  Examples:

@example
(poke) [1,2,3] == [1,2,3]
1
(poke) [[1,2],[3,4]] != [[5,6],[7,8]]
1
@end example

Note that the array elements are compared recursively.

@node Array Indexing
@subsection Array Indexing
@cindex indexing, into arrays

Arrays are indexed using the usual notation, providing an index
enclosed between square brackets with @code{[} and @code{]}:

@example
(poke) [1,2,3][0]
1
(poke) [1,2,3][8#B]
2
@end example

@subsubsection Indexing an array by number of element

Indexing by number of element is performed by providing as index an
expression that evaluates to an integral value.  This value is then
implicitly promoted to an unsigned 64-bit integer when needed.

@cindex exceptions
The valid range for the index is @code{[0,@var{n}]} where @var{n} is
the number of elements stored in the array minus one.  If the passed
integer is out of that range, an @code{E_out_of_bounds} exception is
raised:

@example
(poke) [1,2,3][-1]
unhandled out of bounds exception
(poke) [1,2,3][3]
unhandled out of bounds exception
@end example

@subsubsection Indexing an array by offset of element

Poke also provides a way to refer to the element of an array by the
offset where the element appears inside the array.  This is done by
providing as index an expression that evaluates to an offset value.
This value is then implicitly promoted to an offset with magnitude
unsigned 64-bit integer and bit unit when needed.

If there is not element in the indexed array that starts
@emph{exactly} at the given offset then an @code{E_out_of_bounds}
exception is raised.

@node Array Trimming
@subsection Array Trimming

Indexing is used to fetch elements from arrays.  Another operation,
called @dfn{trimming}, allows you to extract a subset of the array, as
another array.

Trims use the following notation, where a range is specified between
square brackets.  The left sides of the range are closed, whereas the
right sides of the range are open:

@example
(poke) [1,2,3][0:2]
[1,2]
(poke) [1,2,3][1:2]
[2]
(poke) [1,2,3][0:3]
[1,2,3]
@end example

If the minimum side of the range is omitted, it is assumed to be zero.
If the maximum side of the range is omitted, it is assumed to be the
length of the trimmed array:

@example
(poke) [1,2,3][:2]
[1,2]
(poke) [1,2,3][1:]
[2,3]
(poke) [1,2,3][:]
[1,2,3]
@end example

The elements of the base array and the trimmed sub-array are copied by
shared value, exactly like when passing arguments to functions.  This
means that for simple types, copies of the elements are done:

@example
(poke) var a = [1,2,3]
(poke) var s = a[1:2]
(poke) s[0] = 66
(poke) a
[1,2,3]
@end example

However, for complex types like arrays and structs, the values are
shared:

@example
(poke) var a = Packet[] @@ 0#B
(poke) var s = a[1:2]
(poke) s[0].field = 66
(poke) a[1].field
66
@end example

There is an alternative syntax that can be used in order to denote
trims, in which the length of the trim is specified instead of the
index of the second element.  It has this form:

@example
(poke) [1,2,3][0+:3]
[1,2,3]
(poke) [1,2,3][1+:2]
[2,3]
(poke) [1,2,3][1+:0]
[]
@end example

@node Array Elements
@subsection Array Elements

The @code{in} operator can be used to determine whether a given
element is stored in an array.  Examples:

@example
(poke) 2 in [1,2,3]
1
(poke) 2 in [4,5,6]
0
(poke) 10#b in [1#B] #! Promotion doesn't happen. !#
<stdin>:1:1: error: invalid operand in expression
<stdin>:1:1: error: expected offset<int<32>,8>, got offset<int<32>,1>
@end example

@node Array Concatenation
@subsection Array Concatenation

The operator @code{+} can be used to build new arrays resulting from
the concatenation of the elements of two arrays given as operands:

@example
(poke) [1,2] + [3,4,5]
[1,2,3,4,5]
@end example

@noindent
The two arrays given as operands shall have the same elements of the
same type.  The resulting array is always unbounded.

@node Array Attributes
@subsection Array Attributes
@cindex attributes, array attributes
The following attributes are defined for array values.

@table @code
@item size
Gives an offset with the storage occupied by the complete array.
Example:

@example
(poke) [1,2,3]'size
96UL#b
@end example
@item length
Gives the number of elements stored in the array.  Example:

@example
(poke) [1,2,3]'length
3
@end example
@item mapped
Boolean indicating whether the array is mapped.  Examples:

@example
(poke) var a = [1,2,3]
(poke) var b = int[3] @@ 0#B
(poke) a'mapped
0
(poke) b'mapped
1
@end example
@item strict
Boolean indicating whether the array is mapped in strict mode.
Examples:

@example
(poke) var a = int[3] @@ 0#B
(poke) a'strict
1
(poke) var a = int[3] @@! 0#B
(poke) a'strict
0
@end example
@item elem (@var{idx})
Element occupying the position @var{idx} in the array.  Note that this
attribute evaluates to a value of type @code{any}.
@item eoffset (@var{idx})
Offset of the element identified by the index @var{idx} in the array.
Examples:

@example
(poke) [1,2,3]'eoffset (1)
32#b
@end example

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@item esize (@var{idx})
Size of the element identified by the index @var{idx} in the array.
Examples:

@example
(poke) ["foo","x",""]'esize (1)
16#b
@end example

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@item ename (@var{idx})
The string @code{"[@var{idx}]"}.

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@end table

@node Arrays as Stacks
@subsection Arrays as Stacks

Array values can be manipulated as stacks using two operators:
@code{apush} and @code{apop}.

The @code{apush} operator gets an array and a value and appends the
value into the array.  Note it has a destructive side-effect in the
array value, and also it returns the modified value:

@example
(poke) var a = [1,2,3]
(poke) apush (a, 4)
[1,2,3,4]
(poke) a
[1,2,3,4]
@end example

the @code{apop} operators gets an array and removes the last value of
the array.  If the given array is empty then @code{E_out_of_bounds} is
raised.  As with @code{apush}, the operation has the side effect of
modifying the given array and also it returns the popped element:

@example
(poke) var a = [1,2,3]
(poke) apop (a)
3
(poke) apop (a)
2
(poke) apop (a)
1
(poke) apop (a)
unhandled out of bounds exception
@end example

@node Structs
@section Structs
@cindex structs

Structs are the main abstraction that Poke provides to structure data.
They contain heterogeneous collections of values.

@menu
* Struct Types::		Simple structs.
* Struct Constructors::		Constructing struct values.
* Struct Comparison::		Comparing struct values
* Field Endianness::            Specifying the endianness of integral fields.
* Accessing Fields::            Getting and fetching the values of fields.
* Field Constraints::		Specifying data integrity.
* Field Initializers::		Initial values for fields.
* Field Labels::		Explicit offsets for fields.
* Pinned Structs::		Fix the offset of fields.
* The OFFSET variable::		Accessing the offset of the current field.
* Integral Structs::		Composite data stored in integers.
* Unions::			Dealing with conditional data.
* Union Constructors::          Constructing union values.
* Integral Unions::		Variants of composite data stored in integers.
* Optional Fields::		Fields that may or may not exist.
* Casting Structs::		Converting structs from one type to another.
* Declarations in Structs::	Declaring stuff within a struct.
* Methods::			Declaring struct methods.
* Computed Fields::             Simulating fields with methods.
* Struct Attributes::		Accessing properties of struct values.
@end menu

@node Struct Types
@subsection Struct Types

A simple struct type definition in Poke looks like:

@example
type Packet =
  struct
  @{
    byte magic;
    uint<16> length;
    byte[length] data;
  @}
@end example

The above defines a ``Packet'', which consists on a magic number
encoded in a byte, a length encoded in an unsigned 16-bit integer, and
an array of @code{length} bytes, which is the payload of the packet.

Each entry in the struct type above defines a @dfn{struct field}.

@cindex fields, anonymous
Each field has a type, which is mandatory, and a name, which is
optional.  Fields without names are not accessible (not even within
the struct itself) but they are handy for expressing esoteric padding.
@xref{Padding and Alignment}.  Example:

@example
type Imm64 =
  struct
  @{
    uint<32> lo;
    uint<32>;
    uint<32> hi;
  @}
@end example

It is not allowed to have several fields with the same name in the
same struct.  The compiler will complain if it finds such an
occurrence.

@node Struct Constructors
@subsection Struct Constructors

Once a struct type gets defined, there are two ways to build struct
values from it.  One is @dfn{mapping}.  @xref{Mapping Structs}.  The
other is using a @dfn{struct constructor}, which is explained in this
section.

Struct constructors have the following form:

@example
@var{type_name} @{ [@var{field_initializer},]@dots{} @}
@end example

@noindent
where each @var{field_initializer} has the form:

@example
[@var{field_name}=]@var{exp}
@end example

Note how each field has an optional name @var{field_name} and a value
@var{exp}.  The expression for the value can be of any type.  For
convenience, a trailing comma is accepted but not required.

Suppose for example that we have a type defined as:

@example
type Packet =
  struct
  @{
    uint<16> flags;
    byte[32] data;
  @}
@end example

We can construct a new packet, with all its fields initialized to
zero, like this:

@example
(poke) Packet @{@}
Packet @{
  flags=0x0UH,
  data=[0x0UB,0x0UB,0x0UB,0x0UB,0x0UB,@dots{}]
@}
@end example

In the constructor we can specify initial values for some of the
fields:

@example
(poke) Packet @{ flags = 0x8 @}
Packet @{
  flags=0x8UH,
  data=[0x0UB,0x0UB,0x0UB,0x0UB,0x0UB,@dots{}]
@}
@end example

It is not allowed to specify initializers that are not part of the
type being constructed:

@example
(poke) Packet @{ foo = 10 @}
<stdin>:1:10: error: invalid struct field `foo' in constructor
Packet @{ foo = 10 @};
         ^~~
@end example

As we shall see later, many struct types define constraints on the
values their fields can hold.  This is a way to implement data
integrity.  While building struct values using constructors, these
constraints are taken into account.  For example, given the following
struct type:

@example
type BPF_Reg =
  struct
  @{
   uint<4> code : code < 11;
  @};
@end example

@noindent
we will get a constraint violation exception if we try to construct an
@code{BPF_Reg}:

@example
(poke) BPF_Reg @{ code = 20 @}
unhandled constraint violation exception
@end example

@node Struct Comparison
@subsection Struct Comparison
@cindex comparing, structs

The equality operator (@code{==}) and the inequality operator
(@code{!=}) can be applied to struct values.  Examples:

@example
(poke) E_eof == E_eof
1
(Poke) Packet @{ payload=0x3 @} != Packet @{ payload=0x4 @}
1
@end example

The struct values are compares recursively.

@node Field Endianness
@subsection Field Endianness

By default fields are accessed in IO space using the current default
endianness.  However, it is possible to annotate fields with an
explicit endianness, like in:

@example
type Foo =
  struct
  @{
    little int a;
    big int b;
    int c;
  @};
@end example

In the example above, the field @code{a} will be stored using
little-endian, the field @code{b} will be stored using big-endian, and
the field @code{c} will be stored using whatever current endianness.

The endianness annotations can be used in any kind of fields, like
offsets, arrays and structs, and will impact the storage ofintegral fields defined in them.

When endianness annotations are used in struct fields which are
themselves structs, they effectively change the default endianness in
the contained struct.  Therefore, in the following struct type:

@example
type Bar =
  struct
  @{
    little struct
    @{
      big int a;
      int b;
    @} f;
    int c;
  @}
@end example

@noindent
@code{a} is big endian, but @code{b} is little endian.

However, note that the scope of the @code{big} and @code{little}
annotations is lexical.  Therefore, in these types:

@example
type Foo =
  struct
  @{
    int a;
    int b;
  @};

type Bar =
  struct
  @{
    big Foo f;
    int c;
  @};
@end example

@noindent
The endianness of the @code{a} and @code{b} fields stored in @code{f}
is the default endianness, not @code{big}.

@node Accessing Fields
@subsection Accessing Fields

Poke uses the usual dot-notation to provide access to struct fields.
Examples:

@example
(poke) var s = struct @{ i = 10, l = 20L @}
(poke) s.i
10
(poke) s.l
20L
@end example

Writing to fields is achieved by having the field reference in the
left side of an assignment statement:

@example
(poke) s.i = 100
(poke) s.l = 200
(poke) s
struct @{i=100,l=200L@}
@end example

@node Field Constraints
@subsection Field Constraints
@cindex constraints
It is common for struct fields to be constrained to their values to
satisfy some conditions.  Obvious examples are magic numbers, and
specification-derived constraints.

In Poke you can specify a field's constraint using the following
syntax:

@example
@var{field_type} @var{field_name} : @var{expression} ;
@end example

@noindent
where @var{expression} is an arbitrary Poke expression, that should
evaluate to an integer value.  The result is interpreted as a boolean.
@cindex ELF
As an example, this is how the ELF magic number is checked for:

@example
type Ctf_Preamble =
  struct
  @{
    uint<16> ctp_magic : ctp_magic == CTF_MAGIC;
    byte ctp_version;
    byte ctp_flags;
  @};
@end example

The constraint expression will often include the field where it is
installed, but that's not mandatory.

Field constraints play an important role in mapping.  On one side,
a map will fail if there is some constraint that fails.  On the other,
they guide the mapping of unbounded arrays.  @xref{Mapping Arrays}.

Another common usage of constraint expressions is to alter the global
state after decoding a field.  For example, this is how mapping an ELF
header sets the current endianness, depending on the value of
@code{ei_data}:

@example
byte ei_data : ei_data == ELFDATA2LSB       \
               ? set_endian (ENDIAN_LITTLE) \
               : set_endian (ENDIAN_BIG);
@end example

Note that @code{set_endian} always returns @code{1}.

@node Field Initializers
@subsection Field Initializers

We saw that field constraints are useful to express magic numbers,
which are pretty common in binary formats.  Imagine a package has a
marker byte at the beginning, which should always be @code{0xff}.  We
could use a constraint like in:

@example
type Packet =
  struct
  @{
    byte marker : marker == 0xff;
    byte length;
    byte[length] payload;
  @};
@end example

This works well when mapping packages.  The constraint is checked and
a constraint violation exception is raised if the first byte of the
alleged package is not @code{0xff}.

However, suppose we want to construct a new @code{Package}, with no
particular contents.  We would use a constructor, but unfortunately:

@example
(poke) Packet @{ @}
unhandled constraint violation exception
@end example

What happened?  Since we didn't specify a value for the marker in the
struct constructor, a default value was used.  The default value for
an integral type is zero, which violates the constraint associated
with the field.  Therefore, we would need to remember to specify the
marker, every time we construct a new packet:

@example
(poke) Packet @{ marker = 0xff @}
Packet @{
  marker=0xffUB,
  length=0x0UB,
  payload=[]
@}
@end example

Unfortunately, such markers and magic numbers are not precisely very
memorable.  To help with this, Poke has the notion of @dfn{type field
initializers}.  Let's use one in our example:

@example
type Packet =
  struct
  @{
    byte marker = 0xff;
    byte length;
    byte[length] payload;
  @};
@end example

Note how the syntax is different than the one used for constraints.
When a field in a struct type has an initializer, the struct
constructor will use the initializer expression as the initial value
for the field.  For example:

@example
(poke) Packet @{@}
Packet @{
  marker=0xffUB,
  length=0x0UB,
  payload=[]
@}
@end example

It is possible to specify both a constraint and an initializer in the
same field.  Suppose we want to support several kinds of packets,
characterized by several markers.  The supported markers are however a
closed set.  We could do it like this:

@example
type Packet =
  struct
  @{
    byte marker = 0xff : marker in [0xffUB, 0xfeUB];
    byte length;
    byte[length] payload;
  @};
@end example

Note however that struct mappers do not make use of field
initializers, since the mapped IO space provides values for all the
fields in the struct type.  If we mapped the @code{Packet} type above,
we would need to add also a constraint to make sure the value of
@code{marker} is the right one.  This also applies to constructing
@code{Packet} types where an initial value is explicitly specified.
We could do it by specifying both an initial value, and a constraint
expression:

@example
type Packet =
  struct
  @{
    byte marker = 0xff : marker == 0xff;
    byte length;
    byte[length] payload;
  @};
@end example

This idiom is to common that Poke provides a more compact syntax to
denote it, that avoids verbosity and replicated logic:

@example
type Packet =
  struct
  @{
    byte marker == 0xff;
    byte length;
    byte[length] payload;
  @};
@end example

It is considered good practice to design struct types in a way that a
constructor with no arguments will result in something usable.

@node Field Labels
@subsection Field Labels

In structs, each field is associated with an offset, which is relative
to the beginning of the struct.  This is the offset used when reading
and writing the field from/to the IO space when mapping.

The offset is reset to zero bits at the beginning of the struct type,
and it is increased by the size of the fields:

@example
struct
@{
                /* Offset: 0#b */
    uint<1> f1; /* Offset: 1#b */
    sint<7> f2; /* Offset: 1#B */
    int     f3; /* Offset: 5#B */
    Bar     f4; /* Offset: 23#B */
@}
@end example

It is possible to specify an alternative offset for a field using a
@dfn{field label}.

Consider for example an entry in an ELF symbol table.  Each entry has
a @code{st_info} field which is 64-bits long, that in turn can be
interpreted as two fields @code{st_bind} and @code{st_type}.

The obvious solution is to encode @code{st_info} as a sub-struct that
is integral, like this:

@example
struct
@{
  elf32_word st_name;
  struct uint<64>
  @{
    uint<60> st_bind;
    uint<4> st_type;
  @} st_info;
@}
@end example

This makes the value of @code{st_info} easily accessible as an
integral value.  But we may prefer to use labels instead:

@example
struct
@{
  elf32_word st_name;
  elf64_word st_info;
  uint<60> st_bind @@ 4#B;
  uint<4> st_type @@ 4#B + 60#b;
@}
@end example

The resulting struct has fields @code{st_info}, @code{st_bind} and
@code{st_type}, with the last two overlapping the first.

Field labels are also useful in the rare cases where a variable part
of a struct depends on some data that comes @emph{after} it.  This
happens for example in the COFF ``line number'' structures, which are
defined like this in C:

@example
struct external_lineno
@{
  union
  @{
    char l_symndx[4]; /* function name symbol index, iff l_lnno == 0*/
    char l_paddr[4];  /* (physical) address of line number */
  @} l_addr;

  char l_lnno[L_LNNO_SIZE]; /* line number  */
@};
@end example

@noindent
@i{i.e.} the @code{l_addr} field is to be interpreted as an index in a
table or as a physical address depending on the value of
@code{l_lnno}.

This is one way to denote the above data structure using Poke struct
labels:

@example
type COFF_LINENO_32 =
  struct
  @{
    uint<32> l_lnno @ 4#B; /* Line number.  */

    union
    @{
      uint<32> l_symndx : l_lnno == 0;
      offset<uint<32>,B> l_paddr;
    @} l_addr @ 0#B;
  @};
@end example

@noindent
Note how the @code{l_lnno} field is stored after @code{l_addr} even if
they appear in reverse order in the struct definition.  This allows us
to refer to @code{l_nno} in the constraints for the union fields.

@node Pinned Structs
@subsection Pinned Structs
@cindex pinned structs
Pinned structs is a convenient way to write struct types where the
offset of all its fields is zero.  They are equivalent to C unions.

For example, consider the @code{_u} field below in a CTF type
description:

@example
type Ctf_Stype_V1 =
  struct
  @{
    Ctf_Name ctt_name;
    Ctf_Info_V1 ctt_info;
    pinned struct
    @{
      uint32 _size;		/* Size of entire type in bytes.  */
      uint32 _type;		/* Reference to another type.  */
    @} _u;
  @};
@end example

Note that integral structs cannot be pinned.  @xref{Integral Structs}.
And field labels are not allowed in pinned structs.

@node The OFFSET variable
@subsection The OFFSET variable
@cindex OFFSET

The offset of the current field being mapped or constructed, relative
to the beginning of the struct, is at all times available in a
variable called @code{OFFSET}.  This is useful in situations where we
need that information in a constraint expression, the boundary of an
array, an initialization value, or the like.

For example, what follows is a typical usage of OFFSET in order to
introduce some padding in a struct:

@example
type Elf64_Note =
  struct
  @{
    [...]

    Elf64_Xword name;
    byte[alignto (OFFSET, 4#B)] padding;

    [...]
  @};
@end example

The value of this variable changes every time a new field is mapped or
constructed, so writing to it will only have effect until the next
field is processed: at that time the value will be overwritten.

@node Integral Structs
@subsection Integral Structs
@cindex integral structs

Integral structs are useful to cover cases where data is stored in
composited integral containers, @i{i.e.} where data is structured
within stored integers.

Basically, when we structure data using Poke structs, arrays and the
like, we often use the same structure than a C programmer would use.
For example, to model ELF RELA structures, which are defined in C
like:

@example
type struct
  @{
    Elf64_Addr   r_offset;  /* Address */
    Elf64_Xword  r_info;    /* Relocation type and symbol index */
    Elf64_Sxword r_addend;  /* Addend */
  @} Elf64_Rela;
@end example

@noindent
we could use something like this in Poke:

@example
type Elf64_Rela =
  struct
  @{
    Elf64_Addr r_offset;
    Elf64_Xword r_info;
    Elf64_Sxword r_addend;
  @};
@end example

Here the Poke struct type is pretty equivalent to the C incarnation.
In both cases the fields are always stored in the given order,
regardless of endianness or any other consideration.

However, there are situations where stored integral values are to be
interpreted as composite data.  This is the case of the @code{r_info}
field above, which is a 64-bit unsigned integer (@code{Elf64_Xword})
which is itself composed by several fields, depicted here:

@example
 63                                          0
+----------------------+----------------------+
|       r_sym          |      r_type          |
+----------------------+----------------------+
 MSB                                       LSB
@end example

In order to support this kind of composition of integers, C
programmers usually resort to either bit masking (most often) or to
the often obscure and undefined behaviour-prone C bit fields.  In the
case of ELF, the GNU implementations define a few macros to access
these @dfn{sub-fields}:

@example
#define ELF64_R_SYM(i)         ((i) >> 32)
#define ELF64_R_TYPE(i)        ((i) & 0xffffffff)
#define ELF64_R_INFO(sym,type) ((((Elf64_Xword) (sym)) << 32) + (type))
@end example

Where @code{ELF64_R_SYM} and @code{ELF64_R_TYPE} are used to extract
the fields from an @code{r_info}, and @code{ELF64_R_INFO} is used to
compose it.  This is typical of C data structures.

We could of course mimic the C implementation in Poke:

@example
fun Elf64_R_Sym = (Elf64_Xword i) uint<32>:
   @{ return i .>> 32; @}
fun Elf64_R_Type = (Elf64_Xword i) uint<32>:
   @{ return i & 0xffff_ffff; @}
fun Elf64_R_Info = (uint<32> sym, uint<32> type) Elf64_Xword:
   @{ return sym as Elf64_Xword <<. 32 + type; @}
@end example

However, this approach has a huge disadvantage: since we are not able
to encode the logic of these "sub-fields" in proper Poke fields, they
become second class citizens, with all that implies: no constraints on
their own, can't be auto-completed, can't be assigned individually,
@i{etc}.

This is where @dfn{integral structs} come to play. These are structs
that are defined exactly like your garden variety Poke structs, with a
small addition:

@example
type Elf64_RelInfo =
  struct uint<64>
  @{
    uint<32> r_sym;
    uint<32> r_type;
  @};
@end example

Note the @code{uint<64>} addition after @code{struct}.  This can be
any integer type (signed or unsigned).  The fields of an integral
struct should be integral themselves (this includes both integers and
offsets) and the total size occupied by the fields should be the same
size than the one declared in the struct's integer type.  This is
checked and enforced by the compiler.

The Elf64 RELA in Poke can then be encoded like:

@example
type Elf64_Rela =
  struct
  @{
    Elf64_Addr r_offset;
    struct Elf64_Xword
    @{
      uint<32> r_sym;
      uint<32> r_type;
    @} r_info;
    Elf64_Sxword r_addend;
  @};
@end example

When an integral struct is mapped from some IO space, the total number
of bytes occupied by the struct is read as a single integer value, and
then the values of the fields are extracted from it.  A similar
process is using when writing.  That is what makes it different with
respect a normal Poke struct.

Consider for example we have the following sequence of bytes in our IO
space (like a file):

@example
0x10 0x20 0x30 0x40  0x50 0x60 0x70 0x80
@end example

Let's see what happens when we map the integral struct above, in both
big and little endian:

@example
(poke) .set endian big
(poke) Elf64_RelInfo @@ 0#B
Elf64_RelInfo @{
  r_sym=0x10203040U,
  r_type=0x50607080U
@}
(poke) .set endian little
(poke) Elf64_RelInfo @@ 0#B
Elf64_RelInfo @{
  r_sym=0x80706050U,
  r_type=0x40302010U
@}
@end example

Looks good.  For comparison, this is what happens when we do the same
with an "equivalent" (not really) non-integral struct operating on the
same data:

@example
type Elf64_RelInfoBogus =
  struct
  @{
    uint<32> r_sym;
    uint<32> r_type;
  @};
@end example

We would get:

@example
(poke) .set endian big
(poke) Elf64_RelInfoBogus @@ 0#B
Elf64_RelInfoBogus @{
  r_sym=0x10203040U,
  r_type=0x50607080U
@}
(poke) .set endian little
(poke) Elf64_RelInfoBogus @ 0#B
Elf64_RelInfoBogus @{
  r_sym=0x40302010U,
  r_type=0x80706050U
@}
@end example

In this case, and unlike with integral structs, the endianness impacts
the bytes of the individual fields, not of the whole struct.

As you can see, integral structs can be used to denote a lot of
commonly found idioms in data structures and this includes a lot of
what is sometimes denoted in C bit field.  However, one should be
cautious when "translating" C structures to Poke, especially when the
C programmer has not been careful and incurres in sometimes obscure
implementation-defined behavior.  An integral struct is not always the
right abstraction to use when we see a C bit field!

As an example of the above, consider the following C struct:

@example
struct regs
@{
 __u8 dst_reg:4;
 __u8 src_reg:4;
@};
@end example

Certain virtual architecture uses that data layout to store registers
in instructions (no comment.)  Thing is, in bit fields like the above
with sub-byte field sizes, the ordering of the fields is not clearly
defined, and ultimately what order to use is up to the compiler,
@i{i.e.} to lore and tradition.  As it happens, GCC encodes
@code{src_reg} in the most significant nibble of the byte and
@code{dst_reg} in the least significant nibble of the byte when
compiling for a little-endian target, and the other way around when
compiling for a little-endian target. (I may have had that wrong, this
always confuses me.)

How could we encode the C struct regs in Poke?  Let's see.

A normal Poke struct clearly won't do it:

@example
type RegsBogus1 =
  struct
  @{
    uint<4> src;
    uint<4> dst;
  @};
@end example

The reason being, the ordering of src and dst does not change when you
switch endianness (since this is Poke, we can in fact talk about real
ordering of bits)... remember, poke is WYPIWIG (what you poke is what
you get) ;)

What about an integral struct?

@example
type RegsBogus2 =
  struct uint<8>
 @{
    uint<4> src;
    uint<4> dst;
 @};
@end example

This won't work either.  In fact, the net effect of the normal
decoding of the normal struct type RegsBogus1 and the
map-an-integer-and-extract-fields decoding of the integral struct
RegsBogus2 is in this case totally equivalent.

A solution is to use a normal struct, and field labels:

@example
type RegsBogus =
  struct
  @{
    var little_p = (get_endian == ENDIAN_LITTLE);

    uint<8> src @@ !little_p * 4#b;
    uint<8> dst @@ little_p * 4#b;
  @};
@end example

At this point, you may be wondering: is there anything particular in a
field defined in an integral struct?  The answer is: no, not at all.
These are regular, first-class fields.  Likewise, integral structs are
perfectly regular structs.  And of course, since this is poke, you can
have integral structs of say, 11 bits, or 3 bits, map them at offsets
not aligned to bytes, and all the typical poke-atrocities that we
enjoy so much.

However, there exist a few restrictions, some of them fundamental, the
others to be lifted eventually:

@itemize @minus
@item
Integral structs can only have integral fields.  This includes
offsets and integral structs.
@item
No labels are allowed in the fields of integral structs.  This is not
a fundamental limitation, and may be supported at some point.
@item
No optional fields are supported in integral structs.  Support for
this is actually partially implemented (the mapper supports them but
not the writer) and most probably will be completed one of these days.
@end itemize

Integral structs can be converted from/to integral structs to/from
integers, so we can do things like:

@example
rel.r_info as uint<64>;
@end example

And also automatic promotions in arithmetic operators, like:

@example
rel.r_info + 20 * rel.r_info.r_type
@end example

Integers can also be ``structured'' into integral structs:

@example
(poke) 0xdeadbeef as Elf32_RelInfo
Elf32_RelInfo @{
  r_sym=(uint<24>) 0xdeadbe,
  r_type=0xefUB
@}
@end example

@node Unions
@subsection Unions
@cindex unions

Union types are defined much like struct types, using a very similar
syntax:

@example
union
@{
  [... elements ...]
@}
@end example

@noindent
Like in structs, the main kind of elements are fields.  However,
unlike in structs, the fields in an union type denote
@dfn{alternatives}.  At any given time, only one alternative is
selected, and therefore union @emph{values} have only one field.

Which of the several alternatives is selected is determined by the
data integrity defined by means of constraint expressions and other
means.  Alternatives are considered in turn, in written order, and the
first alternative that can be used without triggering a constraint
violation exception is selected.

It makes little sense to mark union types as pinned, since whatever
alternative is chosen will always be located at the beginning of the
union.  To avoid unnecesary confusion, the Poke compiler will emit an
error if you try to tag an union as pinned.

Similarly, labels are not allowed in union alternatives.

@node Union Constructors
@subsection Union Constructors

Constructing union values is very similar to constructing struct
values:

@example
@var{type_name} @{ [@var{field_initializer}] @}
@end example

The main difference is that union constructors require either exactly
one field initializer, or none.  Specifying initialization values for
more than one field doesn't make sense in the case of unions.

Suppose we have the following union type:

@example
type Packet =
  union
  @{
    byte b : b > 0;
    int k;
  @};
@end example

If we try to construct a @code{Packet} with an invalid @code{b}, we
get a constraint error:

@example
(poke) Packet @{b = 0@}
unhandled constraint violation exception
@end example

@noindent
The reason why @code{k} is not chosen is because we specified @code{b}
in the union constructor: we explicitly as for a @code{Packet} of that
kind, but the data we provide doesn't accommodate to that.  If we
wanted another kind of value, we could of course specify an initial
value for @code{k} instead:

@example
(poke) Packet @{k = 10@}
Packet @{
  k=10
@}
@end example

If no field initializer is specified in an union constructor then each
alternative is tried assuming all fields have default values, which in
the case of integral types is a zero:

@example
(poke) Packet @{@}
Packet @{
  k=0
@}
@end example

@noindent
Therefore the alternative @code{b} is considered invalid (it has to be
bigger than 0) and @code{k} is chosen instead.

@node Integral Unions
@subsection Integral Unions
@cindex integral unions

Integral unions are useful to cover cases where we have variants of
data encoded in integers.  One example could be CPU instructions
encoded as 16-bit or 32-bit numbers.

@example
type InstructionFormat_1 =
  struct uint<32>
  @{
    uint<20> imm31_12;
    uint<5>  rd;
    uint<7>  opcode : valid_opcode_for_format_1 (opcode);
  @};

type InstructionFormat_2 =
  struct uint<32>
  @{
    uint<7> funct7;
    uint<5> rs2;
    uint<5> rs1;
    uint<3> funct3;
    uint<5>  rd;
    uint<7>  opcode : valid_opcode_for_format_2 (opcode);
  @};

type Instruction =
  union uint<32>
  @{
    InstructionFormat_1 f1;
    InstructionFomrat_2 f2;
  @};
@end example


Like integral structs, integral unions can be casted to/from integers;
e.g., @code{3211411U as Instruction} will create an @code{Instruction}
union.

Like integral structs, integral unions can have fields of the following
types: integral, offset, integral struct or union.

@node Optional Fields
@subsection Optional Fields
@cindex struct fields

Sometimes a field in a struct is optional, @i{i.e.} it exists or not
depending on some criteria.  A good example of this is the ``extended
header'' in a MP3 id3v2 tag.  From the specification:

@example
The second bit (bit 6) indicates whether or not the header is followed
by an extended header. The extended header is described in section
3.2. A set bit indicates the presence of an extended header.
@end example

In order to express this in a Poke struct type, we could of course use
an union, like:

@example
type ID3V2_Tag =
  struct
  @{
    ID3V2_Hdr hdr;
    union
    @{
      ID3V2_Ext_Hdr hdr : hdr.extended_hdr_present;
      struct @{@};
    @} ext;
    @dots{}
  @}
@end example

That's it, we use an union with an empty alternative.  However, this
approach has two problems.  First, it is a bit cumbersome to write and
it becomes necessary to introduce an intermediate named field like
@code{ext} above.  Second, and most importantly, the constraint
expression in the union alternative is evaluated only once the
alternative has been already decoded.

Therefore Poke provides a more convenient way to denote this:

@example
type ID3V2_Tag =
  struct
  @{
     ID3V2_Hdr hdr;
     if (hdr.extended_hdr_present)
       IDV2_Ext_Hdr ext_hdr;
     @dots{}
  @}
@end example

@noindent
In the example above @code{ext_hdr} will be decoded only if
@code{hdr.extended_hdr_present} is true, which is what we want.

Sometimes, however, we really need to decode a field to decide whether
it is present or absent in the final struct.  In these cases, we can
use a suffix syntax:

@example
type ID3V2_Tag =
  struct
  @{
     ID3V2_Hdr hdr;
     IDV2_Ext_Hdr ext_hdr if hdr.extended_hdr_present
                             && ext_hdr.invisible;
     @dots{}
  @}
@end example

@noindent
Note how in this example the field @code{ext_hdr} can appear itself in
the conditional expression.

If both a constraint and a suffix optional field expression are
specified, the second should follow the first.

One important characteristic of optional fields with suffix
expressions to keep in mind is that they are always constructed or
mapped, even when they are omitted.  This is because, generally
speaking, the value of an optional field may be necessary in order to
determine whether the optional field is present or not.

This may result on unexpected results for the unaware pokist, like in
the example below:

@example
struct
@{
  Header hdr;
  byte[hdr.len - hdr.padding] data if hdr.len >= hdr.padding;
@}
@end example

@noindent
where the array index may underflow if @code{hdr.len} is less or equal
than @code{hdr.padding}.

The obvious solution for this problem is to use a prefix conditional
expression instead.

For obvious reasons, optional fields are not allowed in unions.

@node Casting Structs
@subsection Casting Structs

It is possible to cast a struct of some particular type into another
struct type (but not to an union.)  Examples:

@example
(poke) type Foo = struct @{ int i; int j; @};
(poke) type Bar = struct @{ int k; int j; @};
(poke) Bar @{j=2@} as Foo @{@}
Foo @{i=0,j=2@}
@end example

The semantics of the cast are exactly the same than constructing a
struct of the target type using the struct provided as an expression
to the cast.

This means that some fields in the original struct may be ``lost''
when the struct is converted to the new type.  Consider the following
two definitions:

@example
type Foo = struct @{ int i; long j; @};
type Bar = struct @{ int i; @}
@end example

Let's cast some values and see what we obtain:

@example
(poke) Bar @{@}
Bar @{
  i=0x0
@}
(poke) Bar @{@} as Foo
Foo @{
  i=0x0,
  j=0x0L
@}
(poke) Foo @{@} as Bar
Bar @{
  i=0x0
@}
@end example

It is possible to cast an integral struct into an integral type.  The
value to cast is extracted from the struct fields, and then it is
converted to the target type.

Given the following integral struct type:

@example
type Foo =
  struct int<64>
  @{
    int<32> f1;
    uint<32> f2;
  @};
@end example

@noindent
We can operate:

@example
(poke) Foo @{ f1 = 1, f2 = 1 @} as uint<64> + 2
0x100000003UL
@end example

Conversely, it is possible to cast an integral value to an integral struct:

@example
(poke) 0x100000003UL as Foo
Foo @{
  f1=0x1,
  f2=0x3U
@}
@end example

@node Declarations in Structs
@subsection Declarations in Structs

Type declarations, variable declarations and function declarations are
all allowed inside struct and union types.  The scope of the entity
being declared spans from the declaration until the end of the struct
or union type being defined.

Functions can't set fields defined in the struct type.

@node Methods
@subsection Methods

Methods in structs can be defined using the @code{method} keyword in
definitions like this:

@example
method @var{identifier} = @var{function_body}
@end example

@noindent
where @var{function_body} is the body of a function. @xref{Function
Declarations}.

When invoked, a method gets an implicit @emph{last} argument which is
the struct value whose method is being used.

Methods can refer to the fields of the containing struct type that
have been defined before the method itself, by name.

Additionally, the containing struct can be referred explicitly under
the name @code{SELF} in the body of the method.  @code{SELF} has type
@code{any}.

@node Computed Fields
@subsection Computed Fields
@cindex fields, computed

A @dfn{computed field} declaration in a struct or union type has the form:

@example
computed @var{field_type} @var{field_name};
@end example

@noindent
Where @var{field_type} is a type specifier denoting the type of the
computed field, and @var{field_name} is an identifier.  Note that,
unlike regular fields, computed fields cannot have endianness
annotations, constraints defined for them, cannot be optional, nor
have initialization values.

Computed fields are not really stored in the corresponding struct
values: their value is computed every time they are referred, by
invoking a couple of methods that the user should define:

@itemize @minus
@item
A @strong{getter} method with name @code{get_@var{field_name}} and
signature @code{()@var{field_type}}.

This getter is used to compute the value of the computed field when it
is referenced as a r-value.
@item
A @strong{setter} method with name @code{set_@var{field_name}} and
signature @code{(@var{field_type})void}.

This setter is called when the computed field is assigned to in a
l-value.
@end itemize

For example, consider the following struct denoting a RISC-V
instruction where the immediate is distributed in several chunks:

@example
type RV32_InsnFmt_J = struct uint<32>
  @{
    uint<1>   imm20;
    uint<10>  imm10_1;
    uint<1>   imm11;
    uint<8>   imm19_12;
    RV_Reg    rd;
    RV_Opcode opcode : opcode in RV_OPCODES_J;

    computed int<32> imm;

    method get_imm = int<32>:
    @{
      return imm20:::imm19_12:::imm11:::imm10_1:::(0 as int<1>) as int<21>;
    @}

    method set_imm = (int<32> imm) void:
    @{
      imm20 = (imm .>> 20) & 1;
      imm10_1 = (imm .>> 19) & 0x3ff;
      imm11 = (imm .>> 10) & 1;
      imm19_12 = (imm .>> 19) & 0xff;
    @}
 @};
@end example

@noindent
When we create a value of type @code{RV32_InsnFmt_J} we can then
externally access the computed field @code{imm} like any other normal
field.  However, computed fields are not accessible in other struct
methods, functions, constraint expressions and the like.

@node Struct Attributes
@subsection Struct Attributes
@cindex attributes, struct attributes
The following attributes are defined for struct values.

@table @code
@item size
Gives an offset with the storage occupied by the complete struct.
Example:

@example
(poke) type Packet = struct @{ byte s; byte[s] d; @}
(poke) (Packet @{ s = 3 @})'size
32UL#b
@end example
@item length
Gives the number of fields stored in the struct.  Note that, due to
absent fields, this doesn't always corresponds to the number of fields
that appear in the definition of a struct type.

For example, for the following struct type:

@example
type Packet =
  struct
  @{
    byte magic: magic in [0xffUB,0xfeUB];
    byte control if magic == 0xfe;
    byte[128] payload;
  @};
@end example

@noindent
We get the following results:

@example
(poke) (Packet @{ magic = 0xff @})'length
2UL
(poke) (Packet @{ magic = 0xfe @})'length
3UL
@end example
@item mapped
Boolean indicating whether the struct is mapped.
@item strict
Boolean indicating whether the struct is mapped in strict mode.
@item elem (@var{idx})
Element occupying the position @var{idx} in the array.  Note that this
attribute evaluates to a value of type @code{any}.
@item eoffset (@var{idx})
Offset of the field identified by the index @var{idx} in the struct.
Examples:

@example
(poke) type Foo = struct @{ int i; long l; @}
(poke) Foo@{@}'eoffset (1)
32#b
@end example

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@item esize (@var{idx})
Size of the field identified by the index @var{idx} in the struct.
Examples:

@example
(poke) type Foo = struct @{ int i; long l; @}
(poke) Foo@{@}'esize (1)
64#b
@end example

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@item ename (@var{idx})
Name of the field identified by the index @var{idx} in the struct.  If
the field is anonymous then the empty string.

Examples:

@example
(poke) type Foo = struct @{ int foo; long bar; @}
(poke) Foo@{@}'ename (0)
"foo"
@end example

If the provided index is out of bounds then @code{E_out_of_bounds} is
raised.
@end table

@node Types
@section Types
@cindex types

@menu
* type::			Naming types.
* The any Type::		Polymorphism.
* The isa Operator::		Testing for types of values.
* typeof::                      Introspection of types.
@end menu

@node type
@subsection @code{type}
@cindex @code{type}
The @code{type} directive allows you to declare named types.  The
syntax is:

@example
type @var{name} = @var{type} [, @var{name} = @var{type}]...;
@end example

@noindent
where @var{name} is the name of the new type, and @var{type} is either
a type specifier or the name of some other type.

The supported type specifiers are:

@table @code
@item int<@var{n}>, uint<@var{n}>
Integral types. @xref{Integer Types}.
@item string
The string type.  @xref{String Types}.
@item @var{type}[@var{boundary}]
Array types.  @xref{Array Types}.
@item struct @{ @dots{} @}
Struct types.  @xref{Struct Types}.
@item (@var{type},@dots{})@var{type}:
Function types.  @xref{Function Types}.
@item any
The @code{any} type is used to implement polymorphism.  @xref{The any Type}.
@cindex any, the @code{any} type
@end table

@node The any Type
@subsection The @code{any} Type
@cindex any, the @code{any} type
@cindex polymorphism
Poke supports polymorphism with the @code{any} type.  This type is
used in contexts where a value of any type is allowed.  For example,
this is how you would declare a function that prints the size of any
given value:

@example
fun print_size = (any value) void:
@{
  printf "%v\n", any'size;
@}
@end example

The rules for handling @code{any} values are simple:
@itemize @minus
@item Everything coerces to @code{any}.
@item Nothing coerces from @code{any}.
@end itemize

This means that using any operator that require certain types with
@code{any} values will fail: you have to cast them first.  Example:

@example
(poke) fun foo = (any v) int: @{ return v as int; @}
@end example

Arrays of @code{any}, @code{any[]}, are also supported:

@node The isa Operator
@subsection The @code{isa} Operator
@cindex isa operator
The binary operator @code{isa} allows you to check for the type of a given
value:

@example
(poke) 10 isa int
1
(poke) "foo" isa string
1
(poke) (Packet @ 0#B) isa Packet
1
(poke) 2 as int<3> isa uint<4>
0
@end example

@node typeof
@subsection @code{typeof}
@cindex typeof

The @code{typeof} language construction provides information about
some Poke type.  It has the form:

@example
typeof (@var{entity})
@end example

@noindent
where @var{entity} is either a value or a type specifier.  It returns
a @code{Pk_Type} value describing either the specified type, or the
type of the specified value.

The @code{Pk_Type} struct provides the following fields for all kind
of types:

@table @code
@item uint<32> code
This is a numerical code that identifies the kind of the described
type.

Valid codes are @code{PK_TYPE_UNKNOWN}, @code{PK_TYPE_INTEGRAL},
@code{PK_TYPE_OFFSET}, @code{PK_TYPE_STRING}, @code{PK_TYPE_ARRAY},
@code{PK_TYPE_STRUCT}, @code{PK_TYPE_FUNCTION} and @code{PK_TYPE_ANY}.

@item string name
If the described type is named, then this field contains the name.
Otherwise it is the empty string.

@example
(poke) type myint = int
(poke) typeof (1 as myint)
Pk_Type @{
  code=1,
  name="myint",
  [...]
@}
@end example

@item int<32> complete_p
If the size of the described type can be determined at compile time,
then this field is 1, 0 otherwise.
@end table

The following fields are provided in @code{Pk_Type} only for integral,
offset and integral struct types:

@table @code
@item int<32> signed_p
If the described type is signed, this field is 1, 0 otherwise.
@item uint<64> size
The number of bits of values of the described integral type, or the
number of bits of the integral value of the described integral struct
type, or the number of bits of the magnitude of the described offset
type.

This would be 23UL for an @code{int<23>} type, and 32UL for an
@code{offset<int<32>,1>} type.
@end table

The following fields are provided in @code{Pk_Type} only for
offset types:

@table @code
@item _unit
The unit of the described offset type, in bit units.  This would be
8UL for an @code{offset<int<32>,B>} type, for example.
@end table

The following fields are provided in @code{Pk_Type} only for array
types:

@table @code
@item int<32> bounded_p
This field is 1 if the described array type is bounded by either
number of elements or by size, 0 otherwise.
@end table

The following fields are provided in @code{Pk_Type} only for struct
types:

@table @code
@item int<32> union_p
This field is 1 if the described struct type is an union, 0 otherwise.
@item int<32> pinned_p
This field is 1 if the described struct type is pinned, 0 otherwise.
@item int<32> integral_p
This field is 1 if the described struct type is integral, 0 otherwise.
@item int<32> nfields
Number of fields defined in the described struct type.  This count
includes optional fields and computed fields.
@item int<32> nmethods
Number of methods defined in the described struct type.
@item string[] fnames
Array of @code{nfields} strings, with the names of the fields defined
in the described struct type.  Anonymous fields feature empty strings.
@item int<32>[] fcomputed
Array of @code{nfields} integers, which are 1 for fields that are
computed, 0 for fields that are not computed.
@item string[] ftypes
Array of @code{nfields} strings, with the type specifiers of the types
of the fields defined in the described struct type.
@item string[] mnames
Array of @code{nmethods} strings, with the names of the methods
defined in the described struct type.
@item string[] mtypes
Array of @code{nmethods} strings, with the type specifiers of the
types of the methods defined in the described struct type.
@item (any)any deintegrator
Closure that, when passed an integral value of the same signedness and
width than the described integral struct, returns the struct value
corresponding to that value.

@example
(poke) type Foo = struct int<32> @{ int<16> a; uint<16> b; @}
(poke) typeof (Foo).deintegrator (0x0eadbeef)
Foo @{
  a=0x0eadH,
  b=0xbeefUH
@}
@end example
@end table

The following fields are provided in @code{Pk_Type} only for struct
and array types:

@table @code
@item (int<32>,int<32>,uint<64>)any mapper
Closure that, when passed an integer predicate denoting whether to do
a strict mapping, an IO space identifier, a bit-offset in the given IO
space, map a value of the described type.

@item (any)void writer
Closure that, when passed a mapped value of the described type, writes
the contents of the value to the corresponding IO space.

@item (any)any integrator
Closure that, when passed a value of the described type (an array type
or an integral struct type) returns the integral value corresponding
to the contents of the value.

@example
(poke) type Foo = struct int<32> @{ int<16> a; uint<16> b; @}
(poke) typeof (Foo).integrator (Foo @{ a = 0xab, b = 0xcd @})
0x00ab00cd
@end example

@item (any,any)int<32> comparator
Closure that, when passed two values of the described type, returns 1
if the values are equal,0 otherwise.
@end table

@node Assignments
@section Assignments
@cindex assignment

@menu
* Simple Assignments::          Changing the value of variables.
* Bit Assignments::             Bit-concatenation expressions in l-value.
@end menu

@node Simple Assignments
@subsection Simple Assignments

The assignment statement has the form:

@example
@var{lvalue} = @var{exp};
@end example

@noindent
where @var{lvalue} is either:

@itemize @bullet
@item A variable.
@item A field reference like @code{foo.bar}.
@item An index reference like @code{foo[30]}.
@item A map of a type, like @code{int @@ 0#B} or @code{int[3] @@ 0#B}
or @code{Packet @@ 12#B}.
@end itemize

In all cases, the type of @var{exp} should match the type of the
referred entity.

Examples:

@example
(poke) foo = 10
(poke) packet.length = 4
(poke) packet.data = [1,2,3,4]
(poke) packet.data[2] = 666
(poke) int @@ 23#B = 23
(poke) string @@ str.offset = "foo"
(poke) int[2] @@ 23#B = [1,2]
(poke) Packet @@ 0#B = Packet @{ @dots{} @}
@end example

@node Bit Assignments
@subsection Bit Assignments

@dfn{Bit assignment} statements have the form:

@example
@var{bconc_exp} = @var{exp}
@end example

@noindent
Where @var{bconc_exp} is an expression composed by bit-concatenation
operators and operands that should be valid l-values themselves, and
@var{exp} is an expression whose type must be integral and whose width
must correspond exactly to the width of the integral type of
@var{bconc_exp}.

For example, consider this code:

@example
var a = 0UH;
var b = 0UH;

a:::b = 0xbeef;
@end example

The assignment above updates the value of the variable @code{a} to be
@code{0xbeUH} and the value of the variable @code{b} to be
@code{0xefUH}.  Note how the signedness of @var{bconc_exp} and
@var{exp} may differ.

Any valid l-value can be used as an operand of the bit-concatenation
operators in @var{bconc_exp}.  For example:

@example
var a = [0UH,0UH];

a[0]:::a[1] = 0xbeef;
@end example

@noindent
or, using struct fields:

@example
type Foo = struct @{ uint<16> hi; uint<16> lo; @};

var f = Foo @{@};
f.hi:::f.lo = 0xbeef;
@end example

@node Compound Statements
@section Compound Statements
@cindex compound statements
@cindex statements, compound statements
Compound statements have the form

@example
@{ @var{stmt@dots{}} @}
@end example

@noindent
where @var{stmt@dots{}} is a list of statements, which can be themselves
compound statements.  Compound statements are primarily used to
sequence instructions like:

@example
@{
  do_a;
  do_b;
  do_c;
@}
@end example

A compound statement introduces a new lexical scope.  Declarations in
the compound statements are local to that statement.

Finally, compound statements can be empty: @code{@{ @}}.

@node Conditionals
@section Conditionals

Poke provides several conditional statements, and a ternary
conditional operator, which are discussed in the sections below.

@menu
* if-else::			Simple conditionals.
* Conditional Expression::	Conditionals in expressions.
@end menu

@node if-else
@subsection @code{if-else}
@cindex conditional statements
The @code{if-else} statement has the form

@example
if (@var{exp}) @var{if_stmt} [else @var{else_stmt}]
@end example

@noindent
where @var{exp} is an expression that should evaluate to a boolean
value (@i{i.e.} to an integer), @var{if_stmt} is a statement that will
be executed if @var{exp} holds true, and @var{else_stmt} is a
statement that will be executed if @var{exp} holds false.  The
@code{else} part of the statement is optional.

@node Conditional Expression
@subsection Conditional Expressions
@cindex conditional expressions
@cindex ternary conditional operator
@cindex @code{@?}
@cindex @code{:}

Poke supports a ternary conditional expression that has the form:

@example
@var{condition} ? @var{true_expression} : @var{false_expression}
@end example

@noindent
where @var{condition} is an expression that should evaluate to a
boolean, and @var{true_expression} and @var{false_expression} are
expressions that have exactly the same type.

@node Loops
@section Loops
@cindex loops
@cindex flow control
Poke supports several iteration statements, which are discussed in the
sections below.

@menu
* while::		Iterate while a condition holds true.
* for::                 Generalized loop statement.
* for-in::		Iterate over the elements of a container.
@end menu

@node while
@subsection @code{while}
@cindex @code{while}
The @code{while} statement has this form:

@example
while (@var{exp}) @var{stmt}
@end example

@noindent
where @var{exp} is an expression that should evaluate to a boolean
(@i{i.e.} to an integer) and @var{stmt} is an statement that will be
executed until @var{exp} holds false.

@cindex @code{break}
It is possible to leave the loop from within @var{stmt} using the
@code{break} statement.  Example:

@example
while (1)
@{
  [@dots{}]
  if (exit_loop)
    break;
@}
@end example

@cindex @code{continue}
It is also possible to jump to the next iteration of the loop from
within @var{stmt} using the @code{continue} statement.  Example:

@example
while (1)
@{
   [@dots{}]
   if (continue_loop)
     continue;
@}
@end example

@node for
@subsection @code{for}
@cindex @code{for}

The @code{for} statement has this form:

@example
for ([@var{decls}];[@var{expr}];[@var{stmts}]) @var{stmt}
@end example

@noindent
where @var{decls} is an optional declaration or chain of declarations
separated by commas, @var{expr} is an optional boolean expression,
@var{stmts} is an optional comma-separated list of statements, and
@var{stmt} is a statement.

@node for-in
@subsection @code{for-in}
@cindex @code{for-in}
The @code{for-in} statement has this form:

@example
for (@var{formal} in @var{container} [where @var{exp}]) @var{stmt}
@end example

@noindent
where, in each iteration, the name @var{formal} will be associated
with consecutive values of @var{container}, which shall be an
expression evaluating to an array or a string.  @var{formal} is
available in @var{stmt}, which is the statement executed in each
iteration.

If the @var{where} part is specified, then only iteration in which
@var{exp} holds true are processed.  @var{formal} can be referred in
@var{exp}.  Note that this doesn't mean the loop will stop after
processing the first ``not selected'' element.  See the following
example:

@example
(poke) for (c in [1,2,3,4] where c % 2) printf " %v", c
 1 3
@end example

@cindex @code{break}
It is possible to leave the loop from within @var{stmt} using the
@code{break} statement.

@cindex @code{continue}
It is also possible to jump to the next iteration of the loop from
within @var{stmt} using the @code{continue} statement.

@node Expression Statements
@section Expression Statements

@cindex side effects
Poke is one of these languages where there is a clear separation
between @dfn{expressions} and @dfn{statements}.  However, it is often
useful to use an expression in the place of an statement, in order to
benefit from its side effects.

For that purpose Poke allows you to expressions as statements using
the following syntax:

@example
@var{exp};
@end example

The value computed by the expression will be discarded.

@node Functions
@section Functions
@cindex functions

@menu
* Function Declarations::	Writing functions.
* Optional Arguments::		Default values for arguments.
* Variadic Functions::		Functions taking any number of arguments.
* Calling Functions::		Invoking functions.
* Function Types::		Useful for defining interfaces.
* Lambdas::                     Functions in expressions.
* Function Comparison::         Comparing function values.
* Function Attributes::         Accessing properties of function values.
@end menu

@node Function Declarations
@subsection Function Declarations
@cindex declarations, function declarations
A function is declared using the following syntax:
@cindex @code{fun}

@example
fun @var{name} = [(@var{formal},@dots{})] @var{ret_type}:
@{
   @dots{} body @dots{}
@}
@end example

@noindent
where @var{name} is the name of the function, which uses the same
namespace as variables and types and @var{ret_type} is the type of
the value returned by the function.  If the function returns no value
then it is @code{void}.

Each @var{formal} argument has the form:

@example
@var{type} @var{name} [= @var{exp}]
@end example

@noindent
where @var{type} is the type of the formal, @var{name} its name, and
@var{exp} is an optional expression that will be used to initialize
the argument in case it is not specified when the function is called.

The last formal argument can take the form @code{@var{name}@dots{}},
meaning the function is variadic.  @xref{Variadic Functions}.

If the function takes no arguments, it is possible to omit the list of
arguments entirely:

@example
fun hello = void: @{ print "Hello!\n"; @}
@end example

The @code{return} statement is used to return values in functions that
return a value.  Example:

@example
fun gcd = (uint<64> a, uint<64> b) uint<64>:
  @{
   if (b == 0)
     return a;
   else
     return gcd (b, a % b);
  @}
@end example

Note that reaching the end of a non-void function will trigger a
run-time error.

@node Optional Arguments
@subsection Optional Arguments
@cindex arguments
Optional function arguments are specified like:

@example
fun atoi = (string s, int b = 10) long: @{ @dots{} @}
@end example

Which means that if the base argument is not specified when passed to
@code{atoi} then it is initialized to 10.

Optional arguments should not appear before any non-optional argument
in function declarations.  The following is not valid Poke:

@example
fun foo = (int i = 10, int j) int: @{ return i + j; @}
@end example

Note that arguments declared before an optional argument can be used
in its initialization expression.  This is valid Poke:

@example
fun foo = (int n, int[n] array = init_array (n)) void: @{ @dots{} @}
@end example

@node Variadic Functions
@subsection Variadic Functions
@cindex variadic functions
Functions getting an arbitrary number of arguments are denoted like
this:

@example
fun printf (string fmt, args@dots{}) void: @{ @dots{} @}
@end example

The variadic argument shall be the last argument in the function, and
it is of type @code{any[]}.

@node Calling Functions
@subsection Calling Functions
@cindex calling, function calls
To call a function, write its name followed by the arguments in
parentheses.  Examples:

@example
foo (1,2,3)
bar ()
@end example

If the function takes no arguments then it is not necessary to write
the empty list of arguments.  Therefore the following two calls are
equivalent:

@example
bar ()
bar
@end example

If the closure of function that takes no arguments is needed, the call
can be avoided by putting the variable immediately inside parenthesis,
like this:

@example
(poke) (foo)
#<closure>
@end example


There is an alternate syntax that can be used in both expressions and
expression-statements.  This alternate syntax is:

@example
@var{function_name} :@var{arg1} @var{val1}@dots{}
@end example

@noindent
where @var{arg1} is the name of an argument and @var{val1} the value
to pass for that argument.  This is useful when using functions as
commands in the REPL:

@cindex dump
@example
(poke) dump :from 12#B :size 16#B :ascii 0
@end example

Note that the named arguments can appear in any order.  The following
two calls are equivalent:

@example
dump :from 12#B :size 16#B
dump :size 16#B :from 12#B
@end example

If this alternate syntax for function calls is to be used in an
expression, it is necessary to surround it with parenthesis:

@example
2 + (gcd :a 1024 :b 8)
@end example

@node Function Types
@subsection Function Types
@cindex function types
@cindex types, function types
Function types are denoted like:

@example
(@var{type},@dots{})@var{ret_type}:
@end example

@noindent
where @var{type} are the types of the arguments and @var{ret_type} is
the type of the value returned by the function.

Optional arguments are marked with a @code{?} after the type.  For
example, the type of the @code{atoi} function with declaration:

@example
fun atoi = (string s, int b = 10) long: @{ @dots{} @}
@end example

@noindent
is @code{(string,int?)long:}.

If the function has variadic arguments, the position of the variadic
argument in the function type specifier contains @code{@dots{}}.  For
example, the type of a @code{printf} function with declaration:

@example
fun printf (string fmt, args@dots{}) void: @{ @dots{} @}
@end example

@noindent
is @code{(string,@dots{})void:}.

@node Lambdas
@subsection Lambdas

Poke support @dfn{lambda expressions} using the following syntax:

@example
lambda @var{function_specifier}
@end example

@noindent
Where @var{function_specifier} is any function specifier.  Examples:

@example
lambda void: @{@}
lambda (int i) int: @{ return i * 2; @}
@end example

@noindent
Lambdas can be manipulated exactly like any other Poke value, and can
be stored in variables.  Therefore, this is an alternative way of
defining a named function (which can't be recursive for obvious
reasons):

@example
var double = lambda (int i) int: @{ return i * 2 @};
@end example

@node Function Comparison
@subsection Function Comparison

The equality operator (@code{==}) and the inequality operator
(@code{!=}) can be applied to functions.  They evaluate to true if the
operands are the same function value.  Examples:

@example
(poke) lambda void: @{@} == lambda void: @{@}
0
(poke) lambda void: @{@} != lambda void: @{@}
1
(poke) var f = void: @{@}
(poke) f == f
1
(poke) f != f
0
@end example

@node Function Attributes
@subsection Function Attributes
@cindex attributes, function attributes

The following attributes are defined for function values.

@table @code
@item size
Gives an offset @code{0#B}, by convention.
@end table

@node Assembler
@section Assembler
@cindex asm, statement
@cindex asm, expression
@cindex assembler

Poke provides two ways to execute PVM assembler instructions: an
@code{asm} expression and an @code{asm} statement.

@noindent
@dfn{asm statements} take the following form:

@example
asm (@var{template} [: @var{outputs} [: @var{inputs}]])
@end example

@noindent
where @var{template} is the written form of a PVM assembler program,
@var{outputs} a comma separated list of l-values ans @var{inputs} a
comma separated list of expressions.

Before the assembler program in @var{template} is executed, the
@var{inputs} (if any) are pushed on the stack.  After the assembler
program is executed, it is expected to have left the same number of
elements on the stack than @var{outputs} specified.  These values are
then assigned to the outputs, in reverse order.

Example:

@example
var result = 0;
asm ("addi; nip2" : result : 10, 20);
@end example

@noindent
After the execution of the @code{asm} statement above, the value 30 is
stored in the variable @code{result}.

@noindent
@dfn{asm expressions} take the form:

@example
asm @var{type}: (@var{template} [: @var{inputs}])
@end example

@noindent
where @var{type} is a simple type specifier and @var{template} and
@var{inputs} have the same meaning than in @code{asm} statements.  A
single output is always expected to be generated on the stack, of type
@var{type}.

Example:

@example
(poke) asm int: ("addi; nip2" : 10, 20) + 2
32
@end example

In both @code{asm} statements and expressions the compiler generates
code that tries to determine whether the execution of the assembler
code results in the original stack being underflown or overflown.  If
this is found to happen an exception @code{E_stack} is raised.

@code{E_stack} is also raised in case an @code{asm} statement or
expression leaves a non-representable PVM value as an output.  This is
for example the case of @code{PVM_NULL}, which can't be represented in
Poke.

Note however that the above mentioned tests are not perfect, and it is
generally possible to crash the system using @code{asm} statements or
expressions.

@node Endianness
@section Endianness
@cindex endianness

Byte endianness is an important aspect of encoding data.  As a good
binary editor poke provides support for both little and big endian,
and will soon acquire the ability to encode exotic endianness like PDP
endian.  Endianness control is integrated in the Poke language, and is
designed to be easily used in type descriptions.

@menu
* set endian::			dot-command to get and set the endianness.
* Endian in Fields::		setting the endianness of struct fields.
* Endian built-ins::		changing endianness programmatically.
@end menu

@node set endian
@subsection @code{.set endian}
@cindex endianness

GNU poke maintains a global variable that holds the current
endianness.  This is the endianness that will be used when mapping
integers whose types do not specify an explicit endianness.

Like other poke global state, this global variable can be
modified using the @command{.set} dot-command:

@example
.set endian little
.set endian big
.set endian host
@end example

@noindent
The current endianness can be obtained like this:

@example
(poke) .set endian
little
@end example

We can easily see how changing the current endianness indeed impacts
the way integers are mapped:

@example
(poke) dump :from 0#B :size 4#B :ruler 0 :ascii 0
00000000: 8845 4c46
(poke) .set endian little
(poke) int @@ 0#B
0x464c4588
(poke) .set endian big
(poke) int @@ 0#B
0x88454c46
@end example

@node Endian in Fields
@subsection Endian in Fields

It is possible to set the endianness of integral fields in struct type
descriptors.  @xref{Field Endianness}.

@node Endian built-ins
@subsection Endian built-ins

As handy as the @command {.set endian} dot-command may be, it is also
important to be able to change the current endianness programmatically
from a Poke program.  For that purpose, the PKL compiler provides a
couple of built-in functions: @code{get_endian} and @code{set_endian}.

Their definitions, along with the specific supported values, look
like:

@example
var ENDIAN_LITTLE = 0;
var ENDIAN_BIG = 1;

fun get_endian = int: @{ @dots{} @}
fun set_endian = (int endian) int: @{ @dots{} @}
@end example

Accessing the current endianness programmatically is especially useful
in situations where the data being poked features a different
structure, depending on the endianness.

@cindex eBPF
A good (or bad) example of this is the way registers are encoded in
eBPF instructions.  eBPF is the in-kernel virtual machine of Linux,
and features an ISA with ten general-purpose registers.  eBPF
instructions generally use two registers, namely the source register
and the destination register.  Each register is encoded using 4 bits,
and the fields encoding registers are consecutive in the instructions.

Typical.  However, for reasons we won't be discussing here the order
of the source and destination register fields is switched depending on
the endianness.

@noindent
In big-endian systems the order is:

@example
dst:4 src:4
@end example

@noindent
Whereas in little-endian systems the order is:

@example
src:4 dst:4
@end example

In Poke, the obvious way of representing data whose structure depends
on some condition is using an union.  In this case, it could read like
this:

@example
type BPF_Insn_Regs =
  union
  @{
    struct
    @{
      BPF_Reg src;
      BPF_Reg dst;
    @} le : get_endian == ENDIAN_LITTLE;

    struct
    @{
      BPF_Reg dst;
      BPF_Reg src;
    @} be;
  @};
@end example

Note the call to the @code{get_endian} function (which takes no
arguments and thus can be called Algol68-style, without specifying an
empty argument list) in the constraint of the union alternative.  This
way, the register fields will have the right order corresponding to
the current endianness.

Nifty.  However, there is an ever better way to denote the
structure of these fields.  This is it:

@example
type BPF_Insn_Regs =
  struct
  @{
    var little_p = (get_endian == ENDIAN_LITTLE);

    BPF_Reg src @@ !little_p * 4#b;
    BPF_Reg dst @@ little_p * 4#b;
  @};
@end example

This version, where the ordering of the fields is implemented using
field labels, is not only more compact, but also has the virtue of not
requiring additional ``intermediate'' fields like @code{le} and
@code{be} above.  It also shows how convenient can be to declare
variables inside structs.

@noindent
Let's see it in action:

@example
(poke) BPF_Insn_Regs @@ 1#B
BPF_Insn_Regs @{src=#<%r4>,dst=#<%r5>@}
(poke) .set endian big
(poke) BPF_Insn_Regs @@ 1#B
BPF_Insn_Regs @{src=#<%r5>,dst=#<%r4>@}
@end example

Changing the current endianness in constraint expressions is useful
when dealing with binary formats that specify the endianness of the
data that follows using some sort of tag.  This is the case of ELF,
for example.
@cindex ELF
The first few bytes in an ELF header conform what is known as the
@code{e_ident}.  One of these bytes is called @code{ei_data} and its
value specifies the endianness of the data stored in the ELF file.

@noindent
This is how we handle this in Poke:

@example
fun elf_endian = (int endian) byte:
 @{
   if (endian == ENDIAN_LITTLE)
     return ELFDATA2LSB;
   else
     return ELFDAT2MSB;
 @}

[@dots{}]

type Elf64_Ehdr =
  struct
  @{
    struct
    @{
      byte[4] ei_mag : ei_mag[0] == 0x7fUB
                       && ei_mag[1] == 'E'
                       && ei_mag[2] == 'L'
                       && ei_mag[3] == 'F';
      byte ei_class;
      byte ei_data : (ei_data != ELFDATANONE
                      && set_endian (elf_endian (ei_data)));
      byte ei_version;
      byte ei_osabi;
      byte ei_abiversion;
      byte[6] ei_pad;
      offset<byte,B> ei_nident;
    @} e_ident;

    [@dots{}]
  @};
@end example

Note how @code{set_endian} returns an integer value@dots{}  it is always
@code{1}. This is to facilitate its usage in field constraint
expressions.

@node Mapping
@section Mapping
@cindex mapping
The purpose of poke is to edit @dfn{IO spaces}, which are the files or
devices, or memory areas being edited.  This is achieved by mapping
values.  Mapping is perhaps the most important concept in Poke.

@menu
* IO Spaces::			The underlying entities being edited.
* The Map Operator::		The @@ operator.
* Mapping Simple Types::	Mapping integers, offsets and strings.
* Mapping Structs::		Mapping collections of fields.
* Mapping Arrays::		Mapping sequences of things.
* Mapping Functions::           Orthogonality is important.
* Non-strict Mapping::          Working with incorrect data.
* Auto-Remapping::              Switching to explicit remaps.
* Unmapping::			From mapped values to regular values.
@end menu

@node IO Spaces
@subsection IO Spaces
@cindex IO space

GNU poke supports the abstract notion of @dfn{IO space}, which is an
addressable space of Poke objects: integers, strings, arrays, structs,
@i{etc}.  This underlying storage for the IO spaces (which we call @dfn{IO
devices}) can be heterogeneous: from a file your file system to the
memory of some process.

@menu
* open::			Creating IO spaces.
* opensub::                     IO sub spaces.
* openproc::                    IO proc spaces.
* close::			Destroying IO spaces.
* flush::			Flushing IO spaces.
* get_ios::			Getting the current IO space.
* set_ios::			Setting the current IO space.
* iolist::                      Getting a list of open IO spaces.
* openset::                     Opening and setting combined.
* iosize::			Getting the size of an IO space.
* iohandler::                   Getting the handler string of an IO space.
* ioflags::                     Getting the flags of an IO space.
* IO Space Hooks::              Hooking in common operations on IO spaces.
@end menu

@node open
@subsubsection @code{open}
@cindex opening files
@cindex IO space
The @code{open} builtin allows you to create new IO spaces, by opening an
IO device.  It has the following prototype:

@example
fun open = (string @var{handler}, uint<64> flags = 0) int<32>
@end example

@noindent
where @var{handler} is a string identifying the IO device that will
serve the IO space.  This handler can be:

@table @code
@item *@var{name}*
An auto growing memory buffer.
@item pid://[0-9]+
The process ID of some process.
@item /path/to/file
An either absolute or relative path to a file.
@item nbd://@var{host:port}/@var{export}
@itemx nbd+unix:///@var{export}?socket=@var{/path/to/socket}
A connection to an NBD server. @xref{nbd command}
@end table

@var{flags} is a bitmask that specifies several aspects of the
operation, including the mode in which the IO space is opened.  Its
value is usually built by ORing a set of flags that are provided by
the compiler.  These are:

@table @code
@item IOS_F_READ
The IO space is intended to be read.
@item IOS_F_WRITE
The IO space is intended to be written.
@item IOS_F_CREATE
If the IO device doesn't exist, then create it, usually empty.
@end table

@noindent
Note that the specific behaviour of these flags depend on the nature
of the IO space that is opened. For example a memory buffer is always
truncated at creation by default.

In order to ease the usage of @code{open}, a few pre-made bitmaps are
provided to specify opening @dfn{modes}:

@table @code
@item IOS_M_RDONLY
This is equivalent to @code{IOS_F_READ}.
@item IOS_M_WRONLY
This is equivalent to @code{IOS_F_WRITE}.
@item IOS_M_RDWR
This is equivalent to @code{IOS_F_READ | IOS_F_WRITE}.
@end table

The @code{open} builtin returns a signed 32-bit integer.  This number
will identify the just opened IOS until it gets closed.

If there is a problem opening the specified IO device then @code{open}
will raise an @code{E_no_ios} exception.

@node opensub
@subsubsection @code{opensub}
@cindex @code{opensub}

The @code{opensub} standard function allows you to create IO spaces
that show a sub-region of some other IO space.  The prototype is:

@example
fun opensub = (int<32> @var{ios},
               offset<uint<64>,B> @var{base}, offset<uint<64>,B> @var{size},
               string @var{name} = "",
               uint<64> @var{flags} = 0) int<32>
@end example

@noindent
where @var{ios} is the ID of the base IOS, @var{base} is the offset in
@var{ios} where the sub-range starts and @var{size} is the size of the
range.

The argument @var{name} is a descriptive name of the contents of the
range, and it is empty by default.

The argument @var{flags} is an ORed value of @code{IOS_F_*} flags.  If
no explicit @var{flags} are specified then the sub space inherits the
flags of the underlying base IOS.  If explicit @var{flags} are
provided and they contradict the flags of the underlying IOS then
@code{E_ios_flags} is raised.

Trying to access a sub space whose base IOS has been closed results in
a @code{E_io} exception.

@node openproc
@subsubsection @code{openproc}
@cindex @code{openproc}

The @code{openproc} standard function allows you to create IO spaces
to access the memory of some running program, given its process ID.
The prototype is:

@example
fun openproc = (uint<64> @var{pid}, uint<64> @var{flags} = 0) int<32>
@end example

@noindent
where @var{pid} is the process ID whose memory we want to poke and
@var{flags} is a set of open flags.

@node close
@subsubsection @code{close}
@cindex @code{close}

The @code{close} builtin allows you to destroy IO spaces, closing the
underlying IO device.  The prototype is:

@example
fun close = (int<32> @var{ios}) void
@end example

@noindent
where @var{ios} is some previously created IO space.  All pending data
is written to the underlying IO device.

If the IO space specified to @code{close} doesn't exist then an
@code{E_no_ios} exception is raised.

If errors occurred while closing the IO space, then @code{E_ios}
exception is raised.

@node flush
@subsubsection @code{flush}
@cindex @code{flush}

The builtin @code{flush} performs a ``flushing'' operation on a given
IO space.  The prototype is:

@example
fun flush = (int<32> @var{ios}, offset<uint<64>,1> @var{offset}) void
@end example

Where @var{ios} is the IOS identifier where to perform the flush.  The
semantics associated with the ``flushing'' operation depends on the
kind of IO space:

@itemize @bullet
@item Read-only stream IOS will discard already mapped input up to
@var{offset}.  Any further attempt of mapping data at that area will
cause an @var{E_eof} exception (for Early Of File ;)).
@item Flushing is a no-operation for other kind of IO spaces.
@end itemize

@node get_ios
@subsubsection @code{get_ios}
@cindex @code{get_ios}

GNU poke maintains a @dfn{current IO space}, which is the last created
IO space (this includes IO spaces opened and selected using a
dot-command).  The builtin @code{get_ios} returns this space.  It has
the following prototype:

@example
fun get_ios = int<32>
@end example

If there is no IO space, @code{get_ios} will raise the @code{E_no_ios}
exception.

@node set_ios
@subsubsection @code{set_ios}
@cindex @code{set_ios}

The @code{set_ios} builtin allows you to set a specific IO space as the
new current IO space.  It has the following prototype:

@example
fun set_ios = (int<32> @var{ios}) int<32>
@end example

@noindent
where @var{ios} is the IO space that will become the current IO
space.  If the IO space specified to @code{set_ios} doesn't exist,
@code{E_no_ios} will be raised.

Note that @code{set_ios} always returns @code{1}.  This is to ease its
usage in struct fields constraint expressions.

@node iolist
@subsubsection @code{iolist}
@cindex @code{iolist}

At any time there are zero or more open IO spaces.  The @code{iolist}
builtin returns an array of identifiers of these IO spaces.  It has
the following prototype:

@example
fun iolist = int<32>[]
@end example

@noindent
If no IO space is currently open then @code{iolist} returns the empty
array.  The ordering of the IO spaces in the array is arbitrary.

@node openset
@subsubsection @code{openset}
@cindex @code{openset}

Generally when an IO space is opened with @code{open} it won't
automatically become the current IO space; it will only happens if
there wasn't an IO space previously open.

The standard function @code{openset} is like @code{open} that always
sets the opened IO space as the new current space.  It has the
following prototype:

@example
fun openset = (string @var{handler}, uint<64> @var{flags} = 0) int<32>
@end example

@noindent
Where the meaning of @var{handler}, @var{flags} and the returned value
are exactly like in @code{open}.

@node iosize
@subsubsection @code{iosize}
@cindex @code{iosize}

The @code{iosize} builtin returns the size of a given IO space, as an
offset.  It has the following prototype:

@example
fun iosize = (int<32> ios = get_ios) offset<uint<64>,1>
@end example

If the IO space specified to @code{iosize} doesn't exist,
@code{E_no_ios} will be raised.

@node iohandler
@subsubsection @code{iohandler}
@cindex @code{iohandler}

The @code{iohandler} builtin returns the handler string of some given
IO space.  It has the following prototype:

@example
fun iohandler = (int<32> ios = get_ios) string
@end example

If the IO space specified to @code{iohandler} doesn't exist,
@code{E_no_ios} will be raised.

@node ioflags
@subsubsection @code{ioflags}
@cindex @code{ioflags}

The @code{ioflags} builtin returns the flags active in a given IO
space, encoded in an unsigned 64-bit integer.  It has the following
prototype:

@example
fun ioflags = (int<32> ios = get_ios) uint<64>
@end example

If the IO space specified to @code{ioflags} doesn't exist,
@code{E_no_ios} will be raised.

@node IO Space Hooks
@subsubsection IO Space Hooks
@cindex @code{IOS hooks}

The user can hook in the IO subsystem by installing functions that
will be invoked when certain operations on IO spaces are being
performed.

The functions in @code{ios_open_hook} are invoked once a new IO space
has been opened.

The functions in @code{ios_close_pre_hook} are invoked when an IO
space is about to be closed.  Note that a close_pre hook can prevent
the closing operation to be performed by raising an exception.

The functions in @code{ios_close_hook} are invoked when an IO space
has been closed.

The functions in @code{ios_set_hook} are invoked when an IO space
becomes the current IO space.

All the hooks above get the id of an IO space as arguments:

@example
type IOS_Hook_Fn = (int<32>)void
@end example

The different @code{io*} built-ins can be then used by the handlers in
order to fetch additional properties of the IO space.  For example:

@example
fun my_open_handler = (int<32> ios) void:
@{
  printf ("*** The IO space '%s' (%i32d) has been opened.\n",
          iohandler (ios), ios);
@}
@end example

@example
type IOS_Hook_Close_Fn = (int<32>)void;
@end example

@node The Map Operator
@subsection The Map Operator
@cindex mapping

Poke values reside in memory, and their in-memory representation is
not visible from Poke programs.  For example, @code{32} is a 32-bit
signed integer value, and it happens to not be boxed in the Poke
Virtual Machine.  Therefore, it occupies exactly 32-bit in the memory
of the machine running poke.  Other values, like arrays for example,
are boxed, and they need to store various meta-data.

@cindex @code{@@}
Regardless of the internal representation, we say these values live
``in memory''.  Now, it is also possible to ``map'' a value to some
area in some underlying IO space.  This is done with the map operator
@code{@@}, which has two alternate syntax:

@example
@var{type} @@ @var{offset}
@var{type} @@ @var{ios} : @var{offset}
@end example

The ternary version creates a new value using the data located at the
offset @var{offset} in the specified IO space @var{ios}, which shall
be an expression evaluating to a signed 32-bit integer.

The binary version uses the current IO space.

If there is no IO space, or the specified IO space doesn't exist, an
@code{E_no_ios} exception is raised:

@example
(poke) int @@ 0#B
unhandled no IOS exception
@end example

The value created in a map can be either mapped or not mapped.
Mapping simple types produces not mapped values, whereas mapping
non-simple types create mapped values.

The value attributes @code{mapped} and @code{offset} can be used to
check whether a value is mapped or not, and in that case the offset
where it is mapped:

@example
(poke) var p = Packet @@ 0#B
(poke) p'mapped
0x1
(poke) p'offset
0x0UL#b
@end example

Using the @code{offset} attribute in a not mapped value results in the
@code{E_no_map} exception being raised:

@example
(poke) [1,2,3]'mapped
0x0
(poke) [1,2,3]'offset
unhandled no map exception
@end example

If the type specified in the map is not a simple type, like an array
or a struct, the resulting value is said to be mapped in the IO
space:

@example
(poke) type Packet = struct @{ int i; long l; @}
(poke) Packet @@ 0#B
Packet @{i=0x464c457f,l=0x10102L@}
(poke) uint<8>[2] @@ 0#B
[0x7fUB,0x45UB]
@end example

A very important idea on Poke mapping is that it should be possible to
manipulate mapped and non-mapped values in a transparent way.  For
example, consider the quick sort implementation in poke's standard
library.  The prototype is:

@example
fun qsort = (any[] array, Comparator cmp_f,
               long left = 0, long right = array'length - 1) void
@end example

@noindent
@code{qsort} works with both mapped and not-mapped arrays:

@example
(poke) var a = [2,3,1]
(poke) var b = int[3] @@ 0#B
(poke) b
[1179403647,65794,0]
(poke) qsort (a, IntComparator)
(poke) a
[1,2,3]
(poke) qsort (b, IntComparator)
(poke) b
[0,33620224,1179403647]
(poke) dump :from b'offset :size b'size :ascii 0
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
00000000: 0000 0000 0001 0102 7f45 4c46
@end example

Similarly, you can write functions that operate on abstract
entities and data structures such as ELF relocations and sections,
DWARF DIEs, @i{etc}, and the same code will work with non mapped and
mapped values.

@node Mapping Simple Types
@subsection Mapping Simple Types

Simple values (@i{i.e.} integers, offsets, strings) cannot be mapped.
Therefore, if the type specified in the map is a simple type, the
resulting value will be a regular non-mapped value.  Examples:

@example
(poke) uint<8> @@ 0#B
0x7fUB
(poke) string @@ 0#B
"ELF"
@end example

@node Mapping Structs
@subsection Mapping Structs

Struct and union types can be referred by name in a mapping
(@code{@@}) operation.  For example, given a struct type @code{Foo}
defined as:

@example
type Foo =
  struct
  @{
    int i;
    long l;
  @}
@end example

@noindent
we could get a mapped struct value of that type like this:

@example
var f = Foo @@ 128#B;
@end example

@node Mapping Arrays
@subsection Mapping Arrays

Arrays can be mapped in IO space in three different ways, depending on
the characteristics of the type provided to the mapping operator.

@menu
* Array maps bounded by number of elements::
* Array maps bounded by size::
* Unbounded array maps::
* Mapped bounds in bounded arrays::
@end menu

@node Array maps bounded by number of elements
@subsubsection Array maps bounded by number of elements

When an array type bounded by number of elements is used in a mapping
operation, the resulting mapped array is also bounded by number of
elements.

For example, this is how we would map an array of four 32-bit signed
integers in the current IO space:

@example
(poke) int[4] @@ 0#B
[10,20,30,40]
@end example

Since you can also provide a dynamic array type to the map operator,
the number of elements doesn't need to be constant.  For example,
given the variable @code{nelems} has a value of @code{2}:

@example
(poke) var nelems = 2
(poke) int[nelems + 1] @@ 0#B
[100,222,333]
@end example

If for whatever reason the exact requested number of elements can't be
mapped, then @code{E_constraint} is raised.

@cindex end of file
If an end-of-file condition happens while mapping the array, because
the number of elements specified in the array type, at the given
offset, exceeds the capacity of the underlying IO device, an exception
is raised and the mapping is not completed:

@example
(poke) int[99999999999] @@ 0#B
unhandled EOF exception
@end example

Likewise, if a constraint fails while performing the mapping (while
mapping an array of structs, for example) an exception is raised and
the map is aborted.

@node Array maps bounded by size
@subsubsection Array maps bounded by size

While dealing with binary formats, it often happens that the number of
entities in a collection is given by the space they occupy, rather
than the count itself.

For example, consider ELF sections holding relocations.  These
sections contain a collection of zero or more relocations. The layout
of each relocation is specified by the following type:

@example
type Elf64_Rela =
  struct
  @{
    offset<Elf64_Addr,B> r_offset;
    Elf64_Xword r_info;
    Elf64_Sxword r_addend;
  @};
@end example

The section is described by an entry in the ELF sections header table:

@example
type Elf64_Shdr =
  struct
  @{
    Elf_Word sh_name;
    Elf_Word sh_type;
    Elf64_Xword sh_flags;
    Elf64_Addr sh_addr;
    Elf64_Off sh_offset;
    offset<Elf64_Xword,B> sh_size;
    Elf_Word sh_link;
    Elf_Word sh_info;
    Elf64_Xword sh_addralign;
    offset<Elf64_Xword,b> sh_entsize;
  @};
@end example

The relevant elements of @code{Elf64_Shdr} are @code{sh_offset} and
@code{sh_size}, which indicate the offset of the beginning of the
section's contents, and its size, respectively.  At this point, if we
wanted to get an array with all the relocations in the section, we
could map an array bounded by number of elements like we saw in the
previous section, like this:

@example
(poke) Elf64_Rela[sh_size / 1#Elf64_Rela]
[@dots{} relocs @dots{}]
@end example

However, this approach adoleces from two problems.  First, it doesn't
work with any entity type.  For an offset like @code{1#Elf64_Rela} to
work, it is required to know the size of the type specified as the
unit at compile time.  In the particular case of @code{Elf64_Rela},
that condition is satisfied, but too often that's not the case.  For
example, think about a section containing @code{NULL} terminated
strings: you can't know the number of strings contained in the section
until you actually read it.

Another problem is when the data in the header is corrupt.  Using
the mapping bounded by number of elements, we wouldn't realize it.  It
would be good if the tool would tell us whether the specified size
actually holds an exact number of the requested elements.

A mapping bound by size is what we need.  Fortunately, as we saw when
discussing array types, Poke allows you to specify an offset instead of an
integral value, in the array type specification.  The right amount of
entities (in this case relocations) to strictly satisfy the provided
size will be mapped in the IO space.  So, in order to obtain an array
containing all the relocations in the section, we simply write:

@example
(poke) Elf64_Rela[ehdr.sh_size] @@ ehdr.sh_offset
[@dots{} relocs @dots{}]
@end example

The strictness mentioned above is important.  GNU poke will complain
(and abort the mapping) if it is not possible to map an exact number
of elements.  Thus the following mapping would not be successful:

@example
(poke) int[33#b] @@ 0#B
unhandled out of map bounds exception
@end example

Like in mappings bounded by number of elements, if a constraint fails
while performing the mapping, an exception is raised and the map is
aborted.

Likewise, if the exact size specified as the array boundaries can't be
achieved, then @code{E_constraint} is raised.

@node Unbounded array maps
@subsubsection Unbounded array maps

We mentioned above that if an end-of-file condition happens while
performing a mapping (be it bounded by number of elements or bounded
by size) an EOF exception is raised, and the mapping operation is
aborted.

Unbounded array mappings are performed by using an unbounded array
type in the mapping operation, like in:

@example
@var{type}[] @@ 0#B
@end example

The above construction will map values of type @var{type} in the IO
space until there is an end-of-file condition, or a constraint fails,
whatever happens first.  When it is a constraint expression that
fails, that last element is not included in the mapped array.

Let's assume a binary file contains a series of blocks, located one
after the other, of a kind described by the following struct type:

@example
type Block =
  struct
  @{
     byte magic[2] : magic[0] == 'B' && magic[1] == 'K';
     @dots{} other data @dots{}
  @};
@end example

we can map the blocks using an unbounded array map:

@example
(poke) Block[] @@ 0#B
[ @dots{} blocks @dots{} ]
@end example

If the blocks extend up to the end of the IO space, that many blocks
will be mapped.  If there is some other content in the file following
the blocks, the constraint in the @code{magic} field will fail and
will delimit the map that way (provided the binary format is well
designed.)

@node Mapped bounds in bounded arrays
@subsubsection Mapped bounds in bounded arrays

When an array map is bounded, be it by number of elements or by size,
the bounding value can be mapped itself.  To illustrate how this
works, let's go back to our ELF file and the section containing
relocations.  First, we map an @code{Elf64_Shdr} to get the section
header:

@example
(poke) var shdr = Elf64_Shdr @@ @var{offset}
(poke) shdr.sh_offset
120#B
(poke) shdr.sh_size
24#B
@end example

Now we map an array with the relocations themselves, using a map
bounded by size, as we learned in the last section:

@example
(poke) var relocs = ELF64_Rela[shdr.sh_size] @@ shdr.sh_offset
(poke) relocs'length
3
@end example

Now, observe that @code{shdr.sh_size} is mapped itself!  This means
that, should the section size be modified (to accommodate an extra
relocation, for example) the mapping of @code{relocs} will reflect
that automatically:

@example
(poke) shdr.sh_size = shdr.sh_size + 1#Elf64_Rela
(poke) relocs'length
4
@end example

This is certainly an useful idiom, that is often used while poking
around.  However, sometimes this is @emph{not} what we want.  If we
don't want the mapping bounds of @code{relocs} to be tied to
@code{shdr}, we can just use a temporary for the size:

@example
(poke) var s = shdr.sh_size
(poke) var relocs = Elf64_Rela[s] @@ shdr.sh_offset
@end example

Since simple values (such as the size above) are not mapped, this
trick works as intended.

@node Mapping Functions
@subsection Mapping Functions

Struct fields can be of any type, and that includes function types or
closures.  This is especially useful to build interfaces.

For orthogonality, it is possible to map function types.  The result
of the mapping is not very useful though: mapping a function type
results in a function with an empty body.  If the function returns a
value, a default value (zero for integral types, empty arrays for
array types, @i{etc}) is returned.

Example:

@example
(poke) type Fun = (int) int
(poke) Fun @@ 0#B
#<closure>
(poke) (Fun @@ 0#B) (10)
0x0
@end example

@node Non-strict Mapping
@subsection Non-strict Mapping

The map operator @code{@@} performs what we call a @dfn{strict
mapping}.  This means that the data integrity of the mapped data is
checked for.  Often this is what we want.

However, sometimes we have to work with incorrect or incomplete data.
In these cases, we want to inspect that data and complete it using
poke.

For this purpose poke provides an alternate map operator @code{@@!}
that behaves exactly like the normal operator, but inhibits the
control of data integrity.

The attribute @code{'strict} can be used to query whether a given
value is strict or non-strict.  Non-mapped values are strict by
definition.

@node Auto-Remapping
@subsection Auto-Remapping
@cindex remap

By default poke always tries to handle mapped values in a transparent
way, @i{i.e.} in the same way than if the values weren't mapped.  To
guarantee that the contents of mapped values are always up to date, it
issues @dfn{remaps} every time the value gets referenced.

@noindent
For example, consider the following Poke program:

@example
var arr = X86_Insn[1024];
@end example

@noindent
There, an array of 1024 x86 instructions @code{arr} is mapped.  Every
time the array value is referenced it is re-mapped from the
corresponding IO space.  For example:

@example
arr[2].opcode = 0x0b;
@end example

@noindent
In the reference above, the whole array should be re-mapped from IO
space, because x86 instructions are variable-length, so it is
necessary to re-read all the previous values before accessing the
element in position 2.

This re-mapping 100% guarantees that the mapped values are up to date.
However, this can get very slow, depending on the data structures we
are poking at.

@noindent
poke offers a way to turn this auto re-mapping off.  In Poke code:

@example
vm_set_autoremap (0)
@end example

@noindent
Or using the poke application CLI:

@example
(poke) .set autoremap no
@end example

@noindent
When auto-remap is switched off, mapped values are used as-is.  It is
then the responsibility of the user to explicitly re-map values using
the @code{remap} operator.  Like:

@example
(poke) remap arr
@end example

@node Unmapping
@subsection Unmapping
@cindex unmap

The unary @code{unmap} operator has the form:

@example
unmap @var{value}
@end example

It gets any value and produces the same value, making it not mapped in
case it is a mapped value.

This is useful when we want to read a data structure from the IO space
(say, an array of integers) and then use it for storage without
changing the underlying IO space.  We would do something like:

@example
(poke) var a = unmap (int[3] @@ 10#B)
(poke) a[2] = 100
@end example

@node Exception Handling
@section Exception Handling
@cindex exceptions

Sometimes an error or some other unexpected situation arises.  Poke
provides an exceptions mechanism to deal with these situations.

@menu
* Exceptions::		List of supported exception types.
* try-catch::		Catching exceptions in programs.
* try-until::		Running code until some exception occurs.
* raise::		Raising exceptions in programs.
* exception-predicate:: Turning exceptions into booleans.
* assert::		Asserting conditions in programs.
@end menu

@node Exceptions
@subsection Exceptions

Exceptions in Poke are values of type @code{Exception}, which is a
struct defined like this:

@example
type Exception =
  struct
  @{
    int<32> code;
    string msg;
  @};
@end example

@noindent
where @code{code} identifies the type of the exception, and @code{msg}
is supposed to be a textual description of the exceptional situation.

You can use codes @code{255} and higher for your own exceptions.  For
example:

@example
raise Exception @{ code = 255; msg = "double upset event" @};
@end example

User-defined exceptions should be registered using @code{exception_code}
function.  For example:

@example
var E_my_exception = Exception @{
  code = exception_code,
  msg = "double upset event",
@};
@end example

@noindent
where the @code{E_my_exception.code} is a unique number greater than or
equal to @code{255}.

Exception codes in the range @code{0..254} are reserved for poke.
These are used in predefined exceptions which are standard, and have
specific meanings:

@table @code
@item E_generic
Generic error.
@item E_out_of_bounds
Out of bounds exception.  This can be raised when accessing
containers, like arrays and strings.
@item E_eof
@cindex end of file
End of file exception.  This can be raised when mapping in the IO
space.
@item E_elem
Invalid element exception.  This is raised when attempting to access union
fields that do not exist.
@item E_constraint
Constraint violation exception.  This is raised when a constraint
exception fails while mapping or constructing a struct.
@item E_conv
Conversion exception.  This can be raised while casting values.
@item E_map
No map exception.  This is raised when trying to map a not mapped
value.
@item E_div_by_zero
Division by zero exception.
@item E_no_ios
No IOS exception.  This is raised when the IO space is accessed but
there is no IO space.
@item E_no_return
No return exception.  This is raised when the end of a non-void
function is reached.
@item E_io
Generic IO exception.
@item E_io_flags
Invalid flags were tried while opening an IO device.
@item E_assert
Assertion failure exception.  This is raised by @code{assert} statement.
@item E_overflow
This exception is raised when signed overflow is detected in an
arithmetic operation.
@item E_perm
This exception is raised when some operation can't be performed due to
incorrect (lack of) permissions or capabilities.  An example is
writing to a read-only IO space.
@end table

The exception codes of the standard exceptions are available in the
form of @code{EC_*} variables.  For example, this how you would raise
an IO error with a particular message:

@example
raise Exception @{ code = EC_io,
                   msg = "fluzo capacitator overheat impedes IO" @}
@end example

@node try-catch
@subsection @code{try-catch}
@cindex @code{try-catch}
The @code{try-catch} statement provides a way to catch exceptions and
handle them.

The simplest form of the statement is:

@example
try @var{stmt} catch @var{compound_stmt}
@end example

@noindent
where @var{stmt} is any statement and @var{compound_stmt} is a
compound statement.  First, @var{stmt} is executed.  If during its execution
an exception is raised, then @var{compound_stmt} is executed.

The second form of the statement allows you to catch just one type of
exception:

@example
try @var{stmt} catch if @var{exp} @var{compound_stmt}
@end example

@noindent
where @var{exp} is an expression that should evaluate to an
@code{Exception}.  The handler @var{compound_stmt} will only be
executed if that specific exception is caught.  Any other exception
will be re-raised.

The third form of the statement is the most generic:

@example
try @var{stmt} catch (Exception @var{formal}) @var{compound_stmt}
@end example

@noindent
where @var{formal} is a formal argument that contains the exception
when @var{compound_stmt} is executed.

@node try-until
@subsection @code{try-until}
@cindex @code{try-until}
The @code{try-until} statement allows you to execute instructions until
some exception is caught.  The syntax is:

@example
try @var{stmt} until @var{exp}
@end example

@noindent
where @var{stmt} is the statement that will be executed repeatedly
until some exception is raised.  If the raised exception has type
@var{exp} then execution continues normally.  @var{exp} should be an
expression that evaluates to an @code{Exception}.

@cindex end of file
This statement is particularly useful for mapping IO spaces until an
@code{eof} condition occurs.  For example, this is how we would
compute with every integer in the current IO space:

@example
var o = 0#B;
try
@{
  compute (int @@ o);
  o = o + 1#B;
@} until E_eof;
@end example

@cindex @code{break}
It is possible to leave the loop from within @var{stmt} using the
@code{break} statement.

@node raise
@subsection @code{raise}
@cindex @code{raise}
In previous sections we saw how exceptions are usually the
side-product of performing certain operations.  For example, a
division by zero.

However, it is also useful to explicitly raise exceptions.  The
@code{raise} statement can be used for that purpose.  Its syntax is:

@example
raise @var{exception};
@end example

@noindent
where @var{exception} is an integer.  This integer can be any number,
but most often is one of the @code{E_*} codes defined in Poke.

@node exception-predicate
@subsection exception-predicate

It is often needed to determine whether the execution of some given
code raised some particular exception.  A possibility is to use a
@code{try-catch} statement like in the following example, a
pretty-printer in an union type:

@example
method _print = void:
@{
  try print "#<%u32d>", simple_value;
  catch if E_elem @{ print "%v\n", complex_value; @}
@}
@end example

This works as intended, but it is somewhat verbose and not that easy
to read for anyone not accustomed to the idiom.  It is better to use
the @dfn{exception predicate} operator, @code{?!}, that has two forms:

@example
@var{exp} ?! @var{exception}
@{ [@var{statements}...] @} ?! @var{exception}
@end example

In the first form, the expression @var{exp} is executed.  If the
execution raises the exception @var{exception} then the result of the
predicate is @code{1} (true), otherwise it is @code{0} (false).  The
value of the expression is discarded.

In the second form, the compound statement passed as the first operand
is executed.  If the exception @var{exception} is raised, then the
result of the predicate is @code{0} (false), otherwise it is @code{1}
(true).

Using exception predicates the pretty-printer above can be written in
a much more readable way:

@example
method _print = void:
@{
  if (simple_value ?! E_elem)
    print "%v\n", complex_value;
  else
    print "#<%u32d>", simple_value;
@}
@end example

@noindent
Or, alternatively (and slightly more efficiently):

@example
method _print = void:
@{
  (@{ print "<%u32d>", simple_value; @} ?! E_elem)
    && print "%v\n", complex_value;
@}
@end example

@node assert
@subsection @code{assert}
@cindex @code{assert}

The @code{assert} statement allows you test if a condition is true,
if not, the program will raise an exception with code @code{EC_assert}.

@example
assert (@var{condition})
assert (@var{condition}, @var{message})
@end example

The optional @var{message} will be part of the @code{msg} field of
raised @code{Exception} to explain the situation.  If no @var{message}
is explicitly specified, @code{msg} is set to the source code of the
expression evaluated as the assert condition.

@example
assert (1 == 1);
assert (0 == 0, ``Zero is equal to zero'');
@end example

@code{assert} is useful for writing unit tests.

@node Terminal
@section Terminal

Poke programs have access to a text-oriented terminal, that can be
used to communicate with the user.  The system provides several
primitive functions to manipulate the terminal, which are described in
the sections below.

@menu
* Terminal Colors::		Foreground and background colors.
* Terminal Styling::		Styling output with classes.
* Terminal Hyperlinks::         Click-able hyperlinks.
@end menu

@node Terminal Colors
@subsection Terminal Colors

At every moment the text terminal is configured to use certain
foreground and background colors.  Initially, these are the
@dfn{default colors}, which depend on the underlying terminal
capabilities, and maybe also on some system wide configuration
setting.

Terminal colors are encoded in triplets of integers signifying RGB
colors, @i{i.e.} levels of beams for red, green and blue.  These
triplets are implemented as values of type @code{int<32>[3]}.

The special value @code{[-1, -1, -1]} means the default color, for
both foreground and background.  There are two predefined variables
for this:

@example
var term_default_color = [-1,-1,-1];
var term_default_bgcolor = [-1,-1,-1];
@end example

Getting and setting the foreground and background colors is performed
using the @code{term_get_color}, @code{term_set_color},
@code{term_get_fgcolor} and @code{term_set_fgcolor} functions, with
signatures:

@example
fun term_get_color = int<32>[3]
fun term_set_color = (int<32>[3] color) void
fun term_get_bgcolor = int<32>[3]
fun term_set_bgcolor = (int<32>[3] color) void
@end example

@node Terminal Styling
@subsection Terminal Styling

Styling of the output sent to the terminal is supported in the form of
@dfn{styling classes}.  Each class is characterized by a name, a
string.

A class is activated by using the function @code{term_begin_class},
and deactivated by using the function @code{term_end_class}.  Several
classes can be activated simultaneously, but they should be
deactivated in reverse order.  It is down to the implementation to
decide how to honor some given class, @i{i.e.} to decide how to
characterize it physically (visually, or audibly, or using an
electrical discharge on the keyboard, or whatever) The class may very
well be ignored all together.  Poke programs should never rely on how
a specific class is implemented.

The @code{term_begin_class} and @code{term_end_class} functions have
the following signatures:

@example
fun term_begin_class = (string class) void
fun term_end_class = (string class) void
@end example

@noindent
Styling classes can overlap, but trying to deactivate them in the
wrong order will cause an error in the form of an @code{E_inval}
exception.

@node Terminal Hyperlinks
@subsection Terminal Hyperlinks

The poke terminal supports terminal hyperlinks.  Each hyperlink is
characterized by an URL, and an ID.  The @code{term_begin_hyperlink}
and @code{term_end_hyperlink} functions have the following signatures:

@example
fun term_begin_hyperlink = (string url, string id) void
fun term_end_hyperlink = void
@end example

@noindent
Hyperlinks can overlap, but trying to end an hyperlink when there is
no t a currently open one will result in an @code{E_generic} exception
raised.

@node Printing
@section Printing

Poke programs can print text to the standard output in two ways:
simple unformatted output, and formatted output.

@menu
* print::			simple output.
* printf::			formatted output.
@end menu

@node print
@subsection @code{print}
@cindex @code{print}
@cindex output

The @code{print} statement prints the given string to the standard
output.  @code{print} outputs text strings verbatim.  It can be
invoked using two alternative syntaxes, which are equivalent:

@example
print (@var{str});
print @var{str};
@end example

@code{print} is simple, but fast.  It is good to use it in simple
cases where the information to print out doesn't require any kind of
formatting and styling.

@node printf
@subsection @code{printf}
@cindex @code{printf}
@cindex formatted output

The @code{printf} statement gets a format string and, optionally, a
list of values to print.  It can be invoked using two alternative
syntaxes, which are equivalent:

@example
printf (@var{fmt}[, @var{value}@dots{}])
printf @var{fmt}[, @var{value}@dots{}]
@end example

The format string @var{fmt} is printed verbatim to the standard
output, but for @dfn{format tags} which start with @code{%}.  These
format tags are interpreted especially.
@cindex tags, format tags
Most of the format tags ``consume'' one of the specified values.
Every value in the list shall be described by a tag, or the compiler
will signal an error.  Likewise, values without a corresponding
describing tag is an error.  These tags are:

@table @code
@item %s
Print the argument as a string.
@item %i@var{bits}(d|x|o|b|c)
Print the argument as a signed integer of size @var{bits}.  The last
letter determines how the argument is printed.
@table @code
@item d
@cindex decimal
Print the integer in decimal.
@item x
@cindex hexadecimal
Print the integer in hexadecimal.
@item o
@cindex octal
Print the integer in octal.
@item b
@cindex binary
Print the integer in binary.
@item c
Print the integer as an ASCII character.  This only works with 8 bit
integers.
@end table
@item %u
Same as @code{%i}, but for unsigned integers.
@item %c
A shorter way to write @code{%u8c}.
@item %f@var{bits}[.@var{precision}]d
@itemx %e@var{bits}[.@var{precision}]d
@itemx %g@var{bits}[.@var{precision}]d
These tags accept integral values, that are cast to an integral value
of width @var{bits} and then interpreted as an IEEE 754 floating-point
number.  Currently, valid values for @var{bits} are 32 and 64.

When using @code{%f} tag, the number will be printed in decimal
notation in the style @code{[-]ddd.ddd}, where the number of digits
after the decimal-point character is equal to the @var{precision}
specification.  The default @var{precision} for @code{%f32} tag
is 7 and for @code{%f64} is 15.

The @code{%e} tag will print the number in the style
@code{[-]d.ddde+-dd} where there is one digit (which is non-zero
if the floating-point number is non-zero) before the decimal-point
character and the number of digits after it is equal the
@var{precision}.  The exponent always contains at least two digits;
if the value is zero, the exponent is 00.

And @code{%g} tag will print the number in style @code{%f} or
@code{%e}.  The @var{precision} specifies the number of the
significant digits.  Style @code{%e} is used if the exponent from its
conversion is less than -4 or greater than or equal to the
@var{precision}.  Trailing zeros are removed from the fractional
part of the result; a decimal point appears only if it is followed
by at least one digit.

@example
(poke) printf ("%f64d\n", 0x40091eb851eb851fUL);
3.140000000000000
(poke) printf ("pi:%e32d\n", 0x4048f5c3U)
pi:3.1400001e+00
@end example

@item %v
Print the value printed representation of the argument, which can be
of any type including complex types like arrays and structs.  This is
similar to the @code{write} operation available in many Lisp systems.

This tag supports an optional numerical argument (restricted to a
single digit) that specifies the maximum depth level when printing
structures.  Example:

@example
(poke) printf ("%1v\n", struct @{ s = struct @{ i = 10 @},  l = 20L @});
struct @{s=struct @{@dots{}@},l=0x14L@}
@end example

By default, the depth level is @code{0}, which means no limit.

This tag also support an optional flag that specifies how the value is
printed.  The supported tags are @code{F} (for @dfn{flat}) and
@code{T} (for @dfn{tree}).  The default is flat.  Example:

@example
(poke) printf ("%Tv\n", struct @{ s = struct @{ i = 10 @},  l = 20L @});
struct @{
  s=struct @{
    i=0xa
  @},
  l=0x14L
@}
(poke) printf ("%Fv\n", struct @{ s = struct @{ i = 10 @},  l = 20L @});
struct @{s=struct @{i=0xa@},l=0x14L@}
@end example

This tag is mainly intended to be used in pretty-printers.
@end table

@cindex styled output
The following format tags do not consume arguments.  They support
emitting styled text using the libtextstyle approach of having styling
classes that you can customize in a @file{.css} file.

@table @code
@item %<@var{classname}:
Start the styling class with name @var{classname}.  The class name
cannot be empty.
@item %>
End the last opened styling class.  All styling classes should be
ended before finishing the format string.
@item %%
Print @code{%} character.
@end table

Note that styling classes can be nested, but all classes should be
ended before finishing the format string.

If you use a @var{name} class, you can define how to style it in the
@file{.css} file (poke installs and uses poke-default.css but you can
set the @code{POKE_STYLE} environment variable to point to another
css) like this:

@example
.NAME @{ text-decoration: blink; color : pink; @}
@end example

Examples:

@example
(poke) printf "This is a NAME: %<NAME:xxx%>"
This is a NAME: xxx
(poke) printf "Name: %<string:%s%> Age: %<integer:%i32d%>, "Jose", 39
Name: Jose Age: 39
@end example

@node Comments
@section Comments
@cindex comments

There are several ways to document your Poke programs: comments of
several types and support for separator characters.

@menu
* Multi-line comments::		C-like comments.
* Single line comments::	C++-like comments.
* Vertical separator::		Using form-feed characters.
@end menu

@node Multi-line comments
@subsection Multi-line comments

Poke supports C-like multi-line, comments, which is text enclosed
between @code{/*} and @code{*/} sequences.  These comments cannot be
nested.

@node Single line comments
@subsection Single line comments

C++-like single line comments are supported.  Everything after the
@code{//} sequence is interpreted as a comment, until the end of the
line or the end of the file, whatever comes first.

@node Vertical separator
@subsection Vertical separator

@cindex @code{^L}
@cindex form feed
Poke ignores form feed characters (ASCII code 12, often visualized as
^L).  In GNU software, this character is traditionally used to
separate conceptually different entities in source files.

@node Modules
@section Modules
@cindex @code{load}
@cindex modules

It is common for pickles to depend on stuff defined in other pickles.
In such cases, the @code{load} language construction can be used to
load pickles from your Poke program.  The syntax is:

@example
load @var{module};
@end example

@noindent
where @var{module} is the name of the pickle to load.

For example, your Poke program may want to access some ELF data
structures.  In that case, we can just do:

@example
/* Pickle to poke the contents of some ELF section.  */

load elf;

/* @dots{} code  @dots{} */
@end example

When asked to open a module, poke assumes it is implemented in a file
named @file{@var{module}.pk}.  In the example above, it will try to
load @file{elf.pk}.

The pickles are searched first in the current working directory.  If
not found, then @file{@var{prefix}/share/poke/@var{module}.pk} is
tried next.

If the environment variable @code{POKEDATADIR} is defined, it replaces
@file{@var{prefix}/share/poke}.  This is mainly intended to test a
poke program before it gets installed in its final location.

Nothing prevents you to load the same pickle twice.  This will work if
the pickle doesn't include definitions, but just executes statements.
Otherwise, you will likely get an error due to trying to define stuff
twice.

There is an alternate syntax of the @code{load} construction that is
useful when the module is implemented in a file whose name doesn't
conform to a Poke identifier.  This happens, for example, when the
file name contains hyphens.  Example:

@example
load "my-pickle.pk";
@end example

Note that if you use this variant of @code{load}, you must specify the
full file name, including whatever extension it uses (usually
@file{.pk}).

@node System
@section System

Poke provides several constructions to interact with the operating
system.  These are described in the subsections below.

@menu
* getenv::		Environment variables.
* rand::		Pseudo-random numbers.
@end menu

@node getenv
@subsection getenv

The @code{getenv} built-in function has the form:

@example
fun getenv (string name) string:
@end example

@noindent
Given the name of an environment variable, it returns a string with
its value.  If the variable is not defined in the environment, then
the @code{E_inval} exception is raised.

@node rand
@subsection rand

The @code{rand} built-in function has the form:

@example
fun rand = (uint<32> seed = 0) int<32>:
@end example

@noindent
It returns a pseudo-random number in the range of the unsigned 32-bit
integers.  If a @var{seed} different to zero is provided, it is used
to initialize a new sequence of pseudo-random numbers (which includes
the one returned.)

The @code{gettime} builtin function can be used in order to seed the
pseudo-random number generator very easily:

@example
srand (gettime[1])
@end example

@node VM
@section VM

The Poke language provides a set of pre-defined functions which can be
used to access and modify properties of the underlying implementation
running the Poke programs.

@menu
* @code{vm_obase}::         Get the output base.
* @code{vm_set_obase}::     Set the output base.
* @code{vm_opprint}::       Get whether pretty-printers are used.
* @code{vm_set_opprint}::   Set whether pretty-printers are used.
* @code{vm_oacutoff}::      Get the cutoff limit for array printing.
* @code{vm_set_oacutoff}::  Set the cutoff limit for array printing.
* @code{vm_odepth}::        Get the maximum output depth level.
* @code{vm_set_odepth}::    Set the maximum output depth level.
* @code{vm_oindent}::       Get the output indentation step.
* @code{vm_set_oindent}::   Set the output indentation step.
* @code{vm_omaps}::         Get whether output maps get printed.
* @code{vm_set_omaps}::     Set whether output maps get printed.
* @code{vm_omode}::         Get the current output mode.
* @code{vm_set_omode}::     Set the current output mode.
* @code{vm_autoremap}::     Get whether re-mapping is on.
* @code{vm_set_autoremap}:: Set whether re-mapping is on.
@end menu

@node @code{vm_obase}
@subsection @code{vm_obase}

The pre-defined function @code{vm_obase} returns the current output
base in use in the underlying VM.  It has the following prototype:

@example
fun vm_obase = int<32>:
@end example

@node @code{vm_set_obase}
@subsection @code{vm_set_obase}

The pre-defined function @code{vm_set_obase} sets the output base in
the underlying VM.  It has the following prototype:

@example
fun vm_set_obase = (int<32> @var{obase}) void:
@end example

@noindent
Where @var{obase} is the output base to set.  The base must be one of
2 (for binary), 8 (for octal), 10 (for decimal) or 16 (for
hexadecimal).  If the provided base is not contained in that set,
@code{vm_set_obase} raises @code{E_inval}.

@node @code{vm_opprint}
@subsection @code{vm_opprint}

The pre-defined function @code{vm_opprint} returns a boolean
indicating whether pretty-printers are used when printing struct
values, or not.  It has the following prototype:

@example
fun vm_opprint = int<32>:
@end example

@node @code{vm_set_opprint}
@subsection @code{vm_set_opprint}

The pre-defined function @code{vm_set_opprint} sets a flag in the
underlying VM, indicating whether pretty-printers are used when
printing struct values.  It has the following prototype:

@example
fun vm_set_opprint = (int<32> @var{pretty_print_p}) void:
@end example

@node @code{vm_oacutoff}
@subsection @code{vm_oacutoff}

The pre-defined function @code{vm_oacutoff} returns an integer
indicating the current cutoff limit used when printing array values.
It has the following prototype:

@example
fun vm_oacutoff = int<32>:
@end example

@node @code{vm_set_oacutoff}
@subsection @code{vm_set_oacutoff}

The pre-defined function @code{vm_set_oacutoff} sets the current
cutoff limit used when printing array values.  It has the following
prototype:

@example
fun vm_set_oacutoff = (int<32> @var{cutoff}) void:
@end example

@noindent
where @var{cutoff} is an integer specifying the maximum number of
array elements that get printed.

@node @code{vm_odepth}
@subsection @code{vm_odepth}

The pre-defined function @code{vm_odepth} returns an integer
indicating the maximum depth level used when printing nested
structures.  It has the following prototype:

@example
fun vm_odepth = int<32>:
@end example

@node @code{vm_set_odepth}
@subsection @code{vm_set_odepth}

The pre-defined function @code{vm_set_odepth} sets the maximum depth
level to be used when printing nested structures.  It has the
following prototype:

@example
fun vm_set_odepth = (int<32> @var{depth_level}) void:
@end example

@noindent
where @var{depth_level} specifies the number of depth levels in nested
structs to consider when printing nested structures.

@node @code{vm_oindent}
@subsection @code{vm_oindent}

The pre-defined function @code{vm_oindent} returns an integer
indicating the indentation step.  It has the following prototype:

@example
fun vm_oindent = int<32>:
@end example

@node @code{vm_set_oindent}
@subsection @code{vm_set_oindent}

The pre-defined function @code{vm_set_oindent} sets the indentation
level used when printing nested structures.  It has the following
prototype:

@example
fun vm_set_oindent = (int<32> @var{step}) void:
@end example

@noindent
where @var{step} specifies how much to indent when printing nested
structures.  This value typically translates into the number of white
spaces (or tabs) used in terminals, but this must not be relied on
since other printing media may interpret it differently.

@node @code{vm_omaps}
@subsection @code{vm_omaps}

The pre-defined function @code{vm_omaps} returns a boolean indicating
whether offset information is included when printing out values.  It
has the following prototype:

@example
fun vm_omaps = int<32>:
@end example

@node @code{vm_set_omaps}
@subsection @code{vm_set_omaps}

The pre-defined function @code{vm_set_omaps} sets whether offset
information is included when printing out values. It has the following
prototype:

@example
fun vm_set_omaps = (int<32> @var{omaps_p}) void:
@end example

@node @code{vm_omode}
@subsection @code{vm_omode}

The pre-defined function @code{vm_omode} returns an integer indicating
the currently enabled output mode.  It has the following prototype:

@example
fun vm_omode = int<32>:
@end example

@noindent
There are currently two supported output modes:

@example
var VM_OMODE_PLAIN = 0;
var VM_OMODE_TREE = 1;
@end example

@node @code{vm_set_omode}
@subsection @code{vm_set_omode}

The pre-defined function @code{vm_set_omode} sets the output mode to
use by default when printing values.  It has the following prototype:

@example
fun vm_set_omode = (int<32> @var{omode}) void:
@end example

@noindent
Where @var{omode} is one of the @code{VM_OMODE_*} values documented
above.

@node @code{vm_autoremap}
@subsection @code{vm_autoremap}

The pre-defined function @code{vm_autoremap} returns a boolean
indicating whether auto-remapping of mapped values is switched on.  It
has the following prototype:

@example
fun vm_autoremap = int<32>:
@end example

@node @code{vm_set_autoremap}
@subsection @code{vm_set_autoremap}

The pre-defined function @code{vm_set_autoremap} sets whether
auto-remapping of mapped values is switched on.  It has the following
protoype:

@example
fun vm_set_autoremap = (int<32)void:
@end example

@node Debugging
@section Debugging

@menu
* __LINE__ and __FILE__::	Error locations in traces.
* strace::                      Printing out PVM stack traces
@end menu

@node __LINE__ and __FILE__
@subsection @code{__LINE__} and @code{__FILE__}
@cindex @code{__FILE__}
@cindex @code{__LINE__}
@cindex location
@cindex tracing
When printing traces it is often useful to include a description of
the location of the trace.  The poke compiler provides two builtins
for this purpose.

@table @code
@item __LINE__
@cindex line number
Expands to an unsigned 64-bit integer containing the current line of
the program being compiled.
@item __FILE__
@cindex file name
Expands to a string with the name of the file currently being compiled.  If the
program is read from the standard input (like in the REPL) then this
is @code{"<stdin>"}.
@end table

@node strace
@subsection strace

The @code{strace} built-in function has the prototype:

@example
fun strace = void:
@end example

@noindent
It prints out the current contents of the PVM stack.  This is
especially useful to debug the code generated by the compiler.

@node The Standard Library
@chapter The Standard Library

GNU poke ships with a standard library for Poke programs that is
available to Poke programs.  The following sections detail the
facilities provided by the library.

@menu
* Standard Integral Types::	int, long and the like.
* Standard Offset Types::	off64 and the like.
* Standard Units::		b, B, Kb and the like.
* Conversion Functions::	catos, atoi, @i{etc}.
* Array Functions::             Functions which deal with arrays.
* String Functions::		Functions which deal with strings.
* Character Functions::         Functions which deal with characters.
* Values Functions::            Functions which deal with values in general.
* IO Space Functions::          Functions that operate on IO spaces.
* Sorting Functions::		qsort.
* CRC Functions::               Cyclic Redundancy Checksums.
* Dates and Times::             Processing and displaying dates and times.
* Offset Functions::            Useful functions that operate on offsets.
* Other Functions::             Miscellany.
@end menu

@node Standard Integral Types
@section Standard Integral Types
@cindex integral types
@cindex types, integral types
The Poke standard library provides the following standard integral
types.

@table @code
@item bit
1-bit unsigned integer.
@item nibble
4-bit unsigned integer.
@item uint8
@itemx byte
@itemx char
8-bit unsigned integer.
@item uint16
@itemx ushort
16-bit unsigned integer.
@item uint32
@itemx uint
32-bit unsigned integer.
@item uint64
@itemx ulong
64-bit unsigned integer.
@item int8
8-bit signed integer.
@item int16
@itemx short
16-bit signed integer.
@item int32
@item int
32-bit signed integer.
@item int64
@item long
64-bit signed integer.
@end table

@node Standard Offset Types
@section Standard Offset Types
@cindex offset types
@cindex types, offset types

The Poke standard library provides the following standard offset
types.

@table @code
@item off64
64-bit signed offset in bits.
@item uoff64
64-bit unsigned offset in bits.
@end table

@node Standard Units
@section Standard Units

The following list of units are defined in the standard library.

@cindex units

@noindent
Base units:

@table @code
@item b
bits.
@item N
nibbles.
@item B
bytes.
@end table

@noindent
Base 10 units:

@table @code
@item Kb
@cindex kilobits
kilo bits (1000 bits.)
@item KB
@cindex kilobytes
Kilo bytes (1000 bytes.)
@item Mb
@cindex megabits
Mega bits.
@item MB
@cindex megabytes
Mega bytes.
@item Gb
@cindex gigabits
Giga bits.
@item GB
@cindex gigabytes
Giga bytes.
@end table

@noindent
Base 2 units:

@table @code
@item Kib
@cindex kibibits
kilo bits (1000 bits.)
@item KiB
@cindex kibibytes
Kilo bytes (1000 bytes.)
@item Mib
@cindex mebibits
Mega bits.
@item MiB
@cindex mebibytes
Mega bytes.
@item Gib
@cindex gibibits
Giga bits.
@item GiB
@cindex gibibytes
Giga bytes.
@end table

@node Conversion Functions
@section Conversion Functions
@cindex conversion functions
The Poke standard library provides the following functions to do
useful conversions.

@menu
* catos::		converting characters arrays into string.
* stoca::		filling character arrays from strings.
* strtoi::		convert numeric prefix to integer.
* atoi::		converting strings to integers.
* ltos::		converting integers to strings.
@end menu

@node catos
@subsection @code{catos}
@cindex @code{catos}
@cindex converting, arrays to strings
It is often useful to convert arrays of characters into strings.  The
standard function @code{catos} provides the following interface:

@example
fun catos = (char[] @var{chars}) string: @{ @dots{} @}
@end example

It builds a string containing the characters in @var{chars}, and
returns it.  Examples:

@example
(poke) catos (['a','b','c'])
"abc"
(poke) catos (['\0'])
""
@end example

Note that if the passed array contains a @code{NULL} character
@code{'\0'} then no further characters are processed.  For example:

@example
(poke) catos (['f','o','o','\0','b','a','r'])
"foo"
@end example

@node stoca
@subsection @code{stoca}
@cindex @code{stoca}
@cindex converting, strings to arrays
Sometimes we want to store strings in character arrays.  The standard
function @code{stoca} provides the following interface:

@example
fun stoca = (string s, char[] ca, char fill = 0) void: @{ @dots{} @}
@end example

It fills the given array @var{ca} with the contents of the string
@var{s}.  If the string doesn't fit in the array, then
@code{E_out_of_bounds} is raised.  If the length of the string is less
than the length of the array, the extra characters are set to the
@var{fill} character, which defaults to @code{\0}.

@node strtoi
@subsection @code{strtoi}
@cindex @code{strtoi}
@cindex converting, strings to integers
The standard function @code{strtoi} provides the following interface:

@example
type Strtoi_Result =
  struct
  @{
    uint<64> @var{off};
    int<64> @var{val};
  @};

fun strtoi = (string @var{str}, int @var{base} = 10, ulong @var{start} = 0) Stroi_Result: @{ @dots{} @}
@end example

It parses a signed integral number in the given @var{base} in the
string @var{str} at position @var{start} and returns its value and its
length in the @var{val} and @var{off} fields of a new
@code{Strtoi_Result} instance.

Specifying a @var{start} that is greater than or equal to the length
of the string causes a @code{E_out_of_bound} exception to be raised.

@cindex base, argument in @command{strtoi}
The accepted values for @var{base} are @code{2}, @code{8}, @code{10}
(the default) and @code{16}.  If any other base is requested an
@code{E_inval} exception is raised.

Note that this function parses an integer but permits more characters
to follow the given integer.  As an example, take a look at parsing
``12foo'' in base 16:

@example
#!!# strtoi ("12foo", 16)
Strtoi_Result @{
  off=3UL,
  val=303L
@}
#!!# strtoi ("12foo", 16, 1)
Strtoi_Result @{
  off=3UL,
  val=47L
@}
@end example

@node atoi
@subsection @code{atoi}
@cindex @code{atoi}
@cindex converting, strings to integers
The standard function @code{atoi} provides the following interface:

@example
fun atoi = (string @var{str}, int @var{base} = 10) long: @{ @dots{} @}
@end example

It parses a signed integral number in the given @var{base} in the
string @var{str} and returns it as a signed 64-bit integer.

The accepted values for @var{base} are the same as for @samp{strtoi}.
@xref{strtoi} for a list of supported bases.

Note that atoi allows for extra information to be stored in @var{str}
after the parsed integer.  Thus, this works:

@example
(poke) atoi ("10foo")
10L
@end example

@node ltos
@subsection @code{ltos}
@cindex @code{ltos}

The @code{ltos} standard function converts a given long integer into
its printed representation in some given base.  It has the following
prototype:

@example
fun ltos = (long @var{i}, uint @var{base} = 10) string:
@end example

@noindent
where @var{i} is the number for which to calculate the printed
representation, and @var{base} is a number between 0 and 16.

@node Array Functions
@section Array Functions
@cindex array functions

The Poke standard library provides the following functions to do work
on arrays:

@menu
* reverse::		Reverse the elements of a given array.
@end menu

@node reverse
@subsection @code{reverse}
@cindex @code{reverse}

The standard function @code{reverse} provides the following interface:

@example
fun reverse = (any[] a) void:
@end example

@noindent
It reverses the elements of the given array.

@node String Functions
@section String Functions
@cindex string functions
The Poke standard library provides the following functions to do
work on strings:

@menu
* ltrim::		Remove leading characters.
* rtrim::		Remove trailing characters.
* strchr::		Locate a character in a string, from the beginning.
* strrchr::		Locate a character in a string, from the end.
* strtok::		String tokenization.
@end menu

@node ltrim
@subsection @code{ltrim}
@cindex @code{ltrim}
@cindex whitespace, trimming
The standard function @code{ltrim} provides the following interface:

@example
fun ltrim = (string s, string cs = " \t") string: @{ @dots{} @}
@end example

It returns a copy of the input string @code{s} with any leading
character that also appears in @var{cs} removed.

@node rtrim
@subsection @code{rtrim}
@cindex @code{rtrim}
@cindex whitespace, trimming
The standard function @code{rtrim} provides the following interface:

@example
fun rtrim = (string s, string cs = " \t") string: @{ @dots{} @}
@end example

It returns a copy of the input string @code{s} with any trailing
character that also appears in @var{cs} removed.

@node strchr
@subsection @code{strchr}
@cindex @code{strchr}
@cindex character, locating in a string
The standard function @code{strchr} provides the following interface:

@example
fun strchr = (string s, uint<8> c) int<32>: @{ @dots{} @}
@end example

It returns the index of the first occurrence of the character @var{c}
in the string @var{s}.  If the character is not found in the string,
this function returns the length of the string.

@node strrchr
@subsection @code{strrchr}
@cindex @code{strrchr}
@cindex character, locating in a string
The standard function @code{strrchr} provides the following interface:

@example
fun strrchr = (string s, uint<8> c) int<32>: @{ @dots{} @}
@end example

It returns the index of the last occurrence of the character @var{c}
in the string @var{s}.  If the character is not found in the string,
this function returns -1.

@node strtok
@subsection @code{strtok}
@cindex @code{strtok}
@cindex string, tokenizing

@code{strtok} is a helper for tokenizing strings.  The synopsis of
this API are:

@example
type String_Tokenizer =
  struct
  @{
    uint<64> i;
    string str;
    computed uint<32> more;

    method get_more = uint<32>: @{ @dots{} @}

    method peek = char: @{ @dots{} @}
    method pop = char: @{ @dots{} @}

    method pop_number = (int<32> @var{base} = 10) int<64>: @{ @dots{} @}

    method popdelim = (string @var{delimiters}) string: @{ @dots{} @}
    method poprdelim = (string @var{delimiters}) string: @{ @dots{} @}
  @}

fun strtok = (string @var{a}) String_Tokenizer: @{ @dots{} @}
@end example

@deftypefun String_Tokenizer strtok (string @var{a})
Creates a new tokenizer for the string @var{a}, initially on the zero
position.
@end deftypefun

@cindex @code{String_Tokenizer}
The members of the @code{String_Tokenizer} class are:

@deftypeivar String_Tokenizer uint<64> i
Offset to the next character to be tokenized, @i{i.e.} to the first
character that has not already been consumed.
@end deftypeivar

@deftypeivar String_Tokenizer string str
The string being tokenized.  This string is never tokenized.
@end deftypeivar

@deftypeivar String_Tokenizer uint<32> more
A read-only computed property whose value is truthy if there's more
characters and falsey otherwise.
@end deftypeivar

@deftypemethod String_Tokenizer char poke ()
Returns the first unread character of the string, but does not advance
the @var{i} offset.

Raises @code{E_out_of_bounds} if at the end of the string.
@end deftypemethod

@deftypemethod String_Tokenizer char peek ()
Returns the first unread character of the string, and advances the
tokenizer.

Raises @code{E_out_of_bounds} if at the end of the string.
@end deftypemethod

@deftypemethod String_Tokenizer int<64> pop_number (int<32> @var{base} = 10)
Returns the number at the start of the string and advances the
tokenizer in the given @var{base}.  The bases that are supported are
the same as for @code{strtoi}.
@xref{strtoi} for a list of supported bases.

Raises @code{E_out_of_bounds} if at the end of the string.
@end deftypemethod

@deftypemethod String_Tokenizer string popdelim (string @var{delim})
Returns the substring up to the first character also present in the
string @var{delim}.  Advances the tokenizer to after the delimiter
character (@i{i.e.} it consumes the delimiter character).

Raises @code{E_out_of_bounds} if at the end of the string.
@end deftypemethod

@deftypemethod String_Tokenizer string poprdelim (string @var{delim})
Returns the substring up to the last character also present in the
string @var{delim}.  Advances the tokenizer to after the delimiter
character (@i{i.e.} it consumes the delimiter character).

Raises @code{E_out_of_bounds} if at the end of the string.
@end deftypemethod

@node Character Functions
@section Character Functions
The Poke standard library provides the following functions to deal
with characters:

@menu
* isdigit::             Is given character a decimal digit?
* isxdigit::            Is given character a hexadecimal digit?
@end menu

@node isdigit

The standard function @code{isdigit} provides the following
interface:

@example
fun isdigit = (uint<8> @var{c}) int<32>
@end example

@noindent
it returns 0 when the given ASCII character @var{c} is not a decimal
digit, otherwise it returns a non-zero value.

@node isxdigit

The standard function @code{isxdigit} provides the following
interface:

@example
fun isxdigit = (uint<8> @var{c}) int<32>
@end example

@noindent
it returns 0 when the given ASCII character @var{c} is not a hexadecimal
digit, otherwise it returns a non-zero value.


@node Values Functions
@section Values Functions
The Poke standard library provides the following functions to handle
values in general:

@menu
* eoffset::		Offset of a value's element by name.
@end menu

@node eoffset
@subsection @code{eoffset}

The standard function @code{eoffset} provides the following interface:

@example
fun eoffset = (any val, string name) offset<uint<64>,b>
@end example

@noindent
It returns the offset of the element having the given name in the
given composite value.  If the value doesn't contain an element with
the given name then the exception @code{E_inval} is raised.

@node IO Space Functions
@section IO Space Functions

@menu
* with_cur_ios::        Executing code on a given IO space.
* with_temp_ios::       Executing code on a temporary IO space.
@end menu

@node with_cur_ios
@subsection @code{with_cur_ios}
@cindex @code{with_cur_ios}

The standard function @code{with_cur_ios} provides the following
interface:

@example
fun with_cur_ios = (int<32> @var{ios},
                    Pk_With_Ios_Fn @var{do} = lambda void:@{@},
                    int<32> @var{endian} = get_endian) void:
@end example

Where @var{ios} is an IO space identifier, @var{do} is a function of
type @code{()void} that defaults to a function that does nothing, and
@var{endian} is the endianness to use by default when accessing
@var{ios}.

When invoked it switches to the given IO space and executes the given
function.  Once the function finishes executing, either by returning
or by the raise of an exception, the current IO space and endianness
are restored to their previous values.

@node with_temp_ios
@subsection @code{with_temp_ios}
@cindex @code{with_temp_ios}

The standard function @code{with_temp_ios} provides the following
interface:

@example
fun with_temp_ios = (string @var{handler} = pk_get_unique_mem_ios_handler,
                     uint<32> @var{flags} = 0,
                     Pk_With_Ios_Fn @var{do} = lambda void:@{@},
                     int<32> @var{endian} = get_endian) void:
@end example

Where @var{handler} is the handler to use for the IO space.

@var{flags} are the flags passed to @code{open} when opening the
temporary IO space.  Defaults to 0, which means to use whatever
opening mode makes more sense depending on the kind of the IO space
and it's permissions.

@var{do} is a function of type @code{()void}.  Defaults to a function
that does nothing.

@var{endian} is the endianness to use by default when accessing the
temporary IO space.  Defaults to the current endianness.

When invoked it opens a IO space, like @code{openset}, but takes care
of calling @code{close} and restoring the original current IO space
and endianness after the lambda returns or raises an exception.

It can be used to deal with files without risk of leaking file
descriptors.

@example
with_cur_ios
  :handler "*buf*"
  :do lambda void:
  @{
    var arr = uint<32>[1] @ 0#B;
    arr[0] = 0x1234;
    printf("%v", byte[4] @ 0#b);
  @}
@end example

@node Sorting Functions
@section Sorting Functions
@cindex sorting
@menu
* qsort::		Sorting arrays with quicksort.
@end menu

@node qsort
@subsection @code{qsort}
@cindex @code{qsort}
@cindex quick sort
The standard function @code{qsort} has the following prototype:

@example
fun qsort = (any[] @var{array}, Comparator @var{cmp_f},
               long @var{left} = 0,
               long @var{right} = array'length - 1) void
@end example

@noindent
where @var{array} is the array to sort, @var{cmp_f} is a comparator
function, @var{left} is the index of the first array element to
include in the sorting, and @var{right} is the index of the last array
element to include in the sorting.  Both @var{left} and @var{right}
are optional, and the default is to cover the whole array.

The comparator function @var{cmp_f} should have the following
prototype:

@example
type Comparator = (any,any)int<32>;
@end example

@noindent
for example:

@example
fun CompareInts = (any a, any b) int<32>:
@{
  var ai = a as int<32>;
  var bi = b as int<32>;

  if (ai == bi)
    return 0;
  else if (ai > bi)
    return 1;
  else
    return -1;
@}
@end example


@node CRC Functions
@section CRC Functions
@cindex CRC
@cindex checksum

Many file formats use checksums of one sort or another.
Therefore you may want to write such a checksum or verify
a checksum in a constrained field. @xref{Field Constraints}.

Some formats use simple additive checksums.  Another common checksum
is the Cyclic Redundancy Checksum (CRC).  The standard function
@code{crc32} calculates the 32 bit CRC defined by ISO-3309.

@example
fun crc32 = (byte[] @var{buf}) uint<32>: @{ @dots{} @}
@end example

@noindent
This function returns the 32 bit CRC for the data contained in the
array @var{buf}.

@node Dates and Times
@section Dates and Times
@cindex date
@cindex time
@cindex @code{Timespec}
@cindex @code{gettimeofday}
@cindex @code{ptime}
@cindex @code{POSIX_Time32}

Often a format encodes a date and time expressed as the number of
seconds since midnight, January@tie{}1st 1970@.  You could map these
simply as integers.  However, the standard library provides two types
@code{POSIX_Time32} and @code{POSIX_Time64} which include
pretty-printers to display the date in a human readable format.  The
definition is:

@example
type POSIX_Time@var{size} = struct
@{
  uint<@var{size}>  seconds;

  method _print = void:
  @{ @dots{} @}
@}
@end example

@noindent
where @var{size} is either 32 or 64@.
When pretty printing is enabled, a mapped value of these types will
display similar to

@example
#<2019-Dec-12 8:54:56>
@end example

@noindent
whereas when pretty printing is not enabled, this example would be displayed as:

@example
POSIX_Time32 @{seconds=1576140896U@}
@end example

@noindent
Note that timestamps of this type do not account for leap seconds and
are agnostic towards timezone.

@cindex @code{ptime}
Occasionally you might wish to print an unmapped timestamp value in a
human readable format.  To do this, you can use the @code{ptime} function,
which is defined as follows:

@example
fun ptime = (uint<64> seconds) void:
@{ @dots{} @}
@end example

This pickle also provides a type @code{Timespec} that can store Unix
times to nanosecond precision, and has the form:

@example
type Timespec = struct
@{
  int<64> sec;
  int<64> nsec;
@};
@end example

The function @code{gettimeofday} uses the @code{gettime} builtin in
order to construct an instance of @code{Timespec}.  For example:

@example
(poke) gettimeofday
Timespec @{
  sec=1604918423L,
  nsec=753492546L
@}
(poke) ptime (gettimeofday.sec)
2020-Nov-9 10:40:36
@end example

@node Offset Functions
@section Offset Functions

@menu
* alignto::             Align an offset to a given alignment.
@end menu

@node alignto
@subsection alignto

The @code{alignto} function has the prototype:

@example
fun alignto = (uoff64 offset, uoff64 to) uoff64:
@{ @dots{} @}
@end example

@noindent
It returns an offset that is the result of aligning the given
@var{offset} to the given alignment @var{to}.

@node Other Functions
@section Other Standard Library Functions

The Poke standard library also provides the following functions that
do not fit into the previous few categories:

@menu
* Version Tools::		GNU poke version string utilities.
@end menu

@node Version Tools
@subsection GNU poke Version String Parsing Utilities
@cindex version comparison
@cindex version parsing
The Poke standard library provides utilities for parsing and comparing
GNU poke versions, to allow you to implement conditional code paths or
to check prerequisites.

@deftypevar string pk_version
Version string of the current instance of GNU poke.
@end deftypevar

Note that these functions are @emph{not} generic to any version
string, as they assume and verify the version strings generated by the
GNU poke build system.

@cindex version format
The GNU poke version string that this family of functions accepts fit
the following template:
@samp{@var{X}.@var{Y}[.@var{Z}][-@var{BRANCH}-@var{NN}][-g@var{XDIGITS}][-dirty]}.
Individually, these components are:

@table @samp
@item @var{X}
The major component of this GNU poke version.  Mandatory.

@item @var{Y}
The minor component of this GNU poke version.  Mandatory.

@item @var{Z}
The subminor component of this GNU poke version.  Optional.

@item @var{BRANCH}
A free-form string signifying the current branch a given GNU poke
build comes from.  Optional.  It has two special values:

@table @samp
@item maint
For maintenance branches.
@item dev
For the @code{master} development branch.
@end table

All values other than @samp{dev} are considered equal and lower than
@samp{dev} by the comparison helper described below..

@item @var{NN}
The number commits on @samp{@var{BRANCH}} since the
@samp{@var{X}.@var{Y}[.@var{Z}]} tag.  Present if and only if
@samp{@var{BRANCH}} is present.  This component is also called the
offset.

@item @var{XDIGITS}
The abbreviated commit hash this build was generated on.  For the
purposes of these functions, it is optional, however, in practice, it
can only appear when @samp{@var{BRANCH}} is also present.  Due to the
@samp{-g} prefix, this component also makes a poke version string
appropriate for Git operations, for example: @code{git show
3.0-dev-22-g2e092}.

@item -dirty
This suffix is added if the Git tree this GNU poke version was built
from is dirty.
@end table

The standard library provides the following structured representation
of GNU poke versions:

@cindex @code{Pk_Version}
@example
type Pk_Version =
  struct
  @{
    uint<8> major;
    uint<8> minor;
    uint<8> subminor;
    string branch;
    uint<32> offset;
    string commit;

    uint<8> has_branch_p;
    uint<8> has_subminor_p;
    uint<8> has_commit_p;
    uint<8> dirty_p;
  @};
@end example

@deftypefun string Pk_Version.to_string ()
Method that returns the printed representation of a version.
@end deftypefun

@deftypefun Pk_Version pk_version_parse (string @var{version})
Parses the string @var{version} into a new instance of
@code{Pk_Version}, or raises @code{E_inval} if the string is invalid.
@end deftypefun

The standard library also provides the following helper function:

@deftypefun int<32> pk_vercmp (any @var{a}, any @var{b})
Returns a number greater than, equal to, or lower than zero if @var{a}
compares greater than, equal to or lower than @var{b}, respectively,
as a GNU poke version.

The type of @var{a} and @var{b} must be either @code{string} or
@code{Pk_Version}, otherwise, this function raises an @code{E_conv}.

This function implements partial ordering, as it's not possible to
distinguish some version numbers as greater or lower than each other,
as some could be seen as ``lateral'' to eachother.  For instance,
consider the strings @samp{3.0-a-11-gabcdef} and
@samp{3.0-b-11-gabcdef}.  They are both 11 commits ahead of the 3.0
release, but they aren't on the same lineage, and so, they can't be
considered as greater or lower than eachother.  In cases like these,
@code{pk_vercmp} will attempt to compare commit numbers as a crude
measure.
@end deftypefun


@c @node Hacking Poke
@c @chapter Hacking Poke

@c @menu
@c * Writing Commands::		Extending poke with new commands.
@c @end menu

@c @node Writing Commands
@c @section Writing Commands

@c XXX

@node The Poke Virtual Machine
@chapter The Poke Virtual Machine

@menu
* PVM Instructions::		Virtual Machine instructions.
@end menu

@node PVM Instructions
@section PVM Instructions

@include pvm-insns.texi

@node Table of ASCII Codes
@appendix Table of ASCII Codes

@multitable @columnfractions .1 .1 .1 .7
@headitem Oct @tab Dec @tab Hex @tab Char
@item 000 @tab 0 @tab 00 @tab NUL '\0' (null character)
@item 001 @tab 1 @tab 01 @tab SOH (start of heading)
@item 002 @tab 2 @tab 02 @tab STX (start of text)
@item 003 @tab 3 @tab 03 @tab ETX (end of text)
@item 004 @tab 4 @tab 04 @tab EOT (end of transmission)
@item 005 @tab 5 @tab 05 @tab ENQ (enquiry)
@item 006 @tab 6 @tab 06 @tab ACK (acknowledge)
@item 007 @tab 7 @tab 07 @tab BEL '\a' (bell)
@item 010 @tab 8 @tab 08 @tab BS  '\b' (backspace)
@item 011 @tab 9 @tab 09 @tab HT  '\t' (horizontal tab)
@item 012 @tab 10 @tab 0A @tab LF  '\n' (new line)
@item 013 @tab 11 @tab 0B @tab VT  '\v' (vertical tab)
@item 014 @tab 12 @tab 0C @tab FF  '\f' (form feed)
@item 015 @tab 13 @tab 0D @tab CR  '\r' (carriage ret)
@item 016 @tab 14 @tab 0E @tab SO  (shift out)
@item 017 @tab 15 @tab 0F @tab SI  (shift in)
@item 020 @tab 16 @tab 10 @tab DLE (data link escape)
@item 021 @tab 17 @tab 11 @tab DC1 (device control 1)
@item 022 @tab 18 @tab 12 @tab DC2 (device control 2)
@item 023 @tab 19 @tab 13 @tab DC3 (device control 3)
@item 024 @tab 20 @tab 14 @tab DC4 (device control 4)
@item 025 @tab 21 @tab 15 @tab NAK (negative ack.)
@item 026 @tab 22 @tab 16 @tab SYN (synchronous idle)
@item 027 @tab 23 @tab 17 @tab ETB (end of trans. blk)
@item 030 @tab 24 @tab 18 @tab CAN (cancel)
@item 031 @tab 25 @tab 19 @tab EM  (end of medium)
@item 032 @tab 26 @tab 1A @tab SUB (substitute)
@item 033 @tab 27 @tab 1B @tab ESC (escape)
@item 034 @tab 28 @tab 1C @tab FS  (file separator)
@item 035 @tab 29 @tab 1D @tab GS  (group separator)
@item 036 @tab 30 @tab 1E @tab RS  (record separator)
@item 037 @tab 31 @tab 1F @tab US  (unit separator)
@item 040 @tab 32 @tab 20 @tab SPACE
@item 041 @tab 33 @tab 21 @tab !
@item 042 @tab 34 @tab 22 @tab "
@item 043 @tab 35 @tab 23 @tab #
@item 044 @tab 36 @tab 24 @tab $
@item 045 @tab 37 @tab 25 @tab %
@item 046 @tab 38 @tab 26 @tab &
@item 047 @tab 39 @tab 27 @tab '
@item 050 @tab 40 @tab 28 @tab (
@item 051 @tab 41 @tab 29 @tab )
@item 052 @tab 42 @tab 2A @tab *
@item 053 @tab 43 @tab 2B @tab +
@item 054 @tab 44 @tab 2C @tab ,
@item 055 @tab 45 @tab 2D @tab -
@item 056 @tab 46 @tab 2E @tab .
@item 057 @tab 47 @tab 2F @tab /
@item 060 @tab 48 @tab 30 @tab 0
@item 061 @tab 49 @tab 31 @tab 1
@item 062 @tab 50 @tab 32 @tab 2
@item 063 @tab 51 @tab 33 @tab 3
@item 064 @tab 52 @tab 34 @tab 4
@item 065 @tab 53 @tab 35 @tab 5
@item 066 @tab 54 @tab 36 @tab 6
@item 067 @tab 55 @tab 37 @tab 7
@item 070 @tab 56 @tab 38 @tab 8
@item 071 @tab 57 @tab 39 @tab 9
@item 072 @tab 58 @tab 3A @tab :
@item 073 @tab 59 @tab 3B @tab ;
@item 074 @tab 60 @tab 3C @tab <
@item 075 @tab 61 @tab 3D @tab =
@item 076 @tab 62 @tab 3E @tab >
@item 077 @tab 63 @tab 3F @tab ?
@item 100 @tab 64 @tab 40 @tab @@
@item 101 @tab 65 @tab 41 @tab A
@item 102 @tab 66 @tab 42 @tab B
@item 103 @tab 67 @tab 43 @tab C
@item 104 @tab 68 @tab 44 @tab D
@item 105 @tab 69 @tab 45 @tab E
@item 106 @tab 70 @tab 46 @tab F
@item 107 @tab 71 @tab 47 @tab G
@item 110 @tab 72 @tab 48 @tab H
@item 111 @tab 73 @tab 49 @tab I
@item 112 @tab 74 @tab 4A @tab J
@item 113 @tab 75 @tab 4B @tab K
@item 114 @tab 76 @tab 4C @tab L
@item 115 @tab 77 @tab 4D @tab M
@item 116 @tab 78 @tab 4E @tab N
@item 117 @tab 79 @tab 4F @tab O
@item 120 @tab 80 @tab 50 @tab P
@item 121 @tab 81 @tab 51 @tab Q
@item 122 @tab 82 @tab 52 @tab R
@item 123 @tab 83 @tab 53 @tab S
@item 124 @tab 84 @tab 54 @tab T
@item 125 @tab 85 @tab 55 @tab U
@item 126 @tab 86 @tab 56 @tab V
@item 127 @tab 87 @tab 57 @tab W
@item 130 @tab 88 @tab 58 @tab X
@item 131 @tab 89 @tab 59 @tab Y
@item 132 @tab 90 @tab 5A @tab Z
@item 133 @tab 91 @tab 5B @tab [
@item 134 @tab 92 @tab 5C @tab \  '\\'
@item 135 @tab 93 @tab 5D @tab ]
@item 136 @tab 94 @tab 5E @tab ^
@item 137 @tab 95 @tab 5F @tab _
@item 140 @tab 96 @tab 60 @tab `
@item 141 @tab 97 @tab 61 @tab a
@item 142 @tab 98 @tab 62 @tab b
@item 143 @tab 99 @tab 63 @tab c
@item 144 @tab 100 @tab 64 @tab d
@item 145 @tab 101 @tab 65 @tab e
@item 146 @tab 102 @tab 66 @tab f
@item 147 @tab 103 @tab 67 @tab g
@item 150 @tab 104 @tab 68 @tab h
@item 151 @tab 105 @tab 69 @tab i
@item 152 @tab 106 @tab 6A @tab j
@item 153 @tab 107 @tab 6B @tab k
@item 154 @tab 108 @tab 6C @tab l
@item 155 @tab 109 @tab 6D @tab m
@item 156 @tab 110 @tab 6E @tab n
@item 157 @tab 111 @tab 6F @tab o
@item 160 @tab 112 @tab 70 @tab p
@item 161 @tab 113 @tab 71 @tab q
@item 162 @tab 114 @tab 72 @tab r
@item 163 @tab 115 @tab 73 @tab s
@item 164 @tab 116 @tab 74 @tab t
@item 165 @tab 117 @tab 75 @tab u
@item 166 @tab 118 @tab 76 @tab v
@item 167 @tab 119 @tab 77 @tab w
@item 170 @tab 120 @tab 78 @tab x
@item 171 @tab 121 @tab 79 @tab y
@item 172 @tab 122 @tab 7A @tab z
@item 173 @tab 123 @tab 7B @tab @{
@item 174 @tab 124 @tab 7C @tab |
@item 175 @tab 125 @tab 7D @tab @}
@item 176 @tab 126 @tab 7E @tab ~
@item 177 @tab 127 @tab 7F @tab DEL
@end multitable


@node GNU Free Documentation License
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License

@include fdl.texi

@node Indices
@appendix Indices

@menu
* Concept Index::               General concept index
* PVM Instruction Index::       Index of all PVM instructions
@end menu

@node Concept Index
@appendixsec Concept Index

@c Merge the findex and vindex into the cpindex.
@syncodeindex fn cp
@syncodeindex vr cp

@printindex cp

@node PVM Instruction Index
@appendixsec PVM Instruction Index

@printindex pv

@bye

@c  LocalWords:  texinfo setfilename texi settitle Pokist afourpaper
@c  LocalWords:  dircategory direntry titlepage vskip filll ifnottex
@c  LocalWords:  insertcopying vm pokerc Structs Endianness catos dfn
@c  LocalWords:  atoi qsort cindex bitpatterns noindent samp
@c  LocalWords:  itemx html init REPL TTY rluserman fun printf PVM
@c  LocalWords:  stdin subcommands disassembler ascii POKEDATADIR FLV
@c  LocalWords:  endian endianness obase expr Booleans uint Bitwise
@c  LocalWords:  emph signedness unary boolean ceil bitwise structs
@c  LocalWords:  BSON type struct dstart IETF BFINAL BTYPE BitData
@c  LocalWords:  sizeof kilobits Pokish B'size B'magnitude B'unit UB
@c  LocalWords:  b'magnitude b'unit fUB unhandled var CTF isa str
@c  LocalWords:  polymorphism lvalue stmt Variadic ret namespace gcd
@c  LocalWords:  variadic args PDP programmatically PKL eBPF dst src
@c  LocalWords:  BPF Insn Regs le ident ei ELFDATA LSB ELFDAT MSB ios
@c  LocalWords:  Ehdr ELFDATANONE osabi abiversion nident Unmapping
@c  LocalWords:  Comparator cmp relocations DIEs nelems
@c  LocalWords:  EOF Rela Addr Xword Sxword Shdr addr addralign fmt
@c  LocalWords:  entsize relocs adoleces libtextstyle classname css
@c  LocalWords:  ushort ulong uoff abc ltrim whitespace rtrim fdl
@c  LocalWords:  quicksort array'length comparator printindex
