5. Expression tokens

There are various properties associated with expression tokens which are used to determine the operations that may be performed upon them.

The syntax for introducing expression tokens is:

exp-token:
	EXP exp-storage : type-name :
	NAT

exp-storage:
	rvalue
	lvalue
	const

Expression tokens can be introduced using either the EXP or NAT token introductions. Expression tokens introduced using NAT are constant value designations of type int i.e. they reference constant integer expressions. All other expression tokens are assumed to be non-constant and are introduced using EXP.

All internal expression token identifiers must reside in the macro name space and this is consequently the default name space for such identifiers. Hence the optional name-space, TAG, should not be present in an EXP token introduction. Although the use of an expression token after it has been introduced is very similar to that of an ordinary identifier, as it resides in the macro name space, it has the additional properties listed below:

In order to make use of tokenised expressions, a new symbol, exp-token-name , has been introduced at translation phase seven of the syntax analysis as defined in the ISO C standard. When an expression token identifier is encountered by the preprocessor, an exp-token-name symbol is passed through to the syntax analyser. An exp-token-name provides information about an expression token in the same way that a typedef-name provides information about a type introduced using a typedef. This symbol can only occur as part of a primary-expression (ISO C standard section 6.3.1) and the expression resulting from the use of exp-token-name will have the type, designation and constancy specified in the token introduction. As an example, consider the pragma:

#pragma token EXP rvalue : int : x#

This introduces a token for an expression which is a value designation of type int with internal and external name x.

Expression tokens can either be defined using #define statements or by using externals. They can also be resolved as a result of applying the type-resolution or assignment-resolution operators (see §7.5). Expression token definitions are subject to the following constraints:

The program below provides two examples of the violation of the second constraint.

#pragma token EXP lvalue : int : i#
extern short k;
#define i 6
#define i k

The expression token i is an object designation of type int. The first violation occurs because the expression, 6, does not designate an object. The second violation is because the type of the token expression, i, is int which cannot be resolved to the type short.

If the exp-token-name refers to an expression that designates a value, then the defining expression is converted, as if by assignment, to the type of the expression token using the assignment-resolution operator (see §7.5). With all other designations the defining expression is left unchanged. In both cases the resulting expression is used as the definition of the expression token. This can subtly alter the semantics of a program. Consider the program:

#pragma token EXP rvalue:long:li#
#define li 6
int f() {
	return sizeof(li);
}

The definition of the token li causes the expression, 6, to be converted to long (this is essential to separate the use of li from its definition). The function, f, then returns sizeof(long). If the token introduction was absent however f would return sizeof(int).

Although they look similar, expression token definitions using #defines are not quite the same as macro definitions. A macro can be defined by any preprocessing tokens which are then computed in phase 3 of translation as defined in the ISO C standard, whereas tokens are defined by assignment-expressions which are computed in phase 7. One of the consequences of this is illustrated by the program below:

#pragma token EXP rvalue:int :X#
#define X M + 3
#define M sizeof(int)
int f(int x)
{
	return (x + X);
}

If the token introduction of X is absent, the program above will compile as, at the time the definition of X is interpreted (when evaluating x + X), both M and X are in scope. When the token introduction is present the compilation will fail as the definition of X, being part of translation phase 7, is interpreted when it is encountered and at this stage M is not defined. This can be rectified by reversing the order of the definitions of X and M or by bracketing the definition of X. i.e.

#define X (M+3)

Conversely consider:

#pragma token EXP rvalue:int:X#
#define M sizeof(int)
#define X M + 3
#undef M
int M(int x) {
	return (x + X);
}

The definition of X is computed on line 3 when M is in scope, not on line 6 where it is used. Token definitions can be used in this way to relieve some of the pressures on name spaces by undefining macros that are only used in token definitions. This facility should be used with care as it may not be a straightforward matter to convert the program back to a conventional C program.

Expression tokens can also be defined by declaring the exp-token-name that references the token to be an object with external linkage e.g.

#pragma token EXP lvalue:int:x#
extern int x;

The semantics of this program are effectively the same as the semantics of:

#pragma token EXP lvalue:int:x#
extern int _x;
#define x _x