Preserve original spellings of extended identifiers.

This patch makes cpplib track the original spellings of extended
identifiers, as well as the canonical UTF-8 version, in order to
follow standard semantics properly without needing a convoluted and
undocumented canonicalization in translation phase 1 (see bug 9449
comments 39-46 regarding such a canonicalization).

The spelling is tracked in cpp_identifier and cpp_macro_arg without
making cpp_token any larger.  The original spelling is used for checks
of duplicate macro definitions, stringizing (see the C++ tests added;
this case is only an issue for C++ not C because C makes it
implementation-defined whether a \ is inserted before the \ of a UCN
in a string or character constant when stringizing, while C++ does
not), pasting (relevant when the result is then stringized for C++)
and when macro definitions are output as text (e.g. for -d options).

Once a macro has been defined, only the original spelling of the
argument names needs keeping in the argument list.  While it is being
defined, however, both spellings are needed: the original one for
subsequent saving for checks of duplicate macro definitions, and the
canonical one which is the node marked specially to generate macro
argument tokens rather than normal identifier tokens.  The buffer that
is used to save the original values of the identifier tokens is
changed so that it stores both those original values and a pointer to
the canonical hash nodes, so that those canonical nodes can be found
when their values need restoring after the macro definition has been
parsed.

I believe this covers the known standards issues in extended
identifiers support (the remaining unimplemented C99 areas in GCC all
being floating-point-related), except for C++ translation of extended
characters to UCNs in phase 1 (which I have no plans to work on).
There are however probably issues left with handling of extended
identifiers in other places, as listed in
<https://gcc.gnu.org/ml/gcc-patches/2014-11/msg00337.html> (those
issues are generally the sort of thing that could be addressed as bugs
outside development stage 1).  (The bulk of the potential issues Zack
was concerned about in 2003-5, that resulted in extended identifiers
being disabled in the absence of -fextended-identifiers, were
effectively eliminated by the audit and fixes I did in 2009, however;
that todo list reflects what was left over after that audit.)

Bootstrapped with no regressions on x86_64-unknown-linux-gnu.

libcpp:
	* include/cpp-id-data.h (struct cpp_macro): Update comment
	regarding parameters.
	* include/cpplib.h (struct cpp_macro_arg, struct cpp_identifier):
	Add spelling fields.
	(struct cpp_token): Update comment on macro_arg.
	* internal.h (_cpp_save_parameter): Add extra argument.
	(_cpp_spell_ident_ucns): New declaration.
	* lex.c (lex_identifier): Add SPELLING argument.  Set *SPELLING to
	original spelling of identifier.
	(_cpp_lex_direct): Update calls to lex_identifier.
	(_cpp_spell_ident_ucns): New function, factored out of
	cpp_spell_token.
	(cpp_spell_token): Adjust FORSTRING argument semantics to return
	original spelling of identifiers.  Use _cpp_spell_ident_ucns in
	!FORSTRING case.
	(_cpp_equiv_tokens): Check spellings of identifiers and macro
	arguments are identical.
	* macro.c (macro_arg_saved_data): New structure.
	(paste_tokens): Use original spellings of identifiers from
	cpp_spell_token.
	(_cpp_save_parameter): Add argument SPELLING.  Save both canonical
	node and its value.
	(parse_params): Update calls to _cpp_save_parameter.
	(lex_expansion_token): Save spelling of macro argument tokens.
	(_cpp_create_definition): Extract canonical node from saved data.
	(cpp_macro_definition): Use UCNs in spelling of macro name.  Use
	original spellings of macro argument tokens and identifiers.
	* traditional.c (scan_parameters): Update call to
	_cpp_save_parameter.

gcc:
	* doc/invoke.texi (-std=c99, -std=c11): Don't refer to corner
	cases of extended identifiers.

gcc/testsuite:
	* g++.dg/cpp/ucnid-2.C, g++.dg/cpp/ucnid-3.C,
	gcc.dg/cpp/ucnid-11.c, gcc.dg/cpp/ucnid-12.c,
	gcc.dg/cpp/ucnid-13.c, gcc.dg/cpp/ucnid-14.c,
	gcc.dg/cpp/ucnid-15.c: New tests.


git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@217202 138bc75d-0d04-0410-961f-82ee72b054a4

1 files changed, 2 insertions, 1 deletions

diff --git a/libcpp/traditional.c b/libcpp/traditional.c
index dfb53787a04..3d40c2f2122 100644
--- a/libcpp/traditional.c
+++ b/libcpp/traditional.c
@@ -959,8 +959,9 @@ scan_parameters (cpp_reader *pfile, cpp_macro *macro)
 
       if (is_idstart (*cur))
 	{
+	  struct cpp_hashnode *id = lex_identifier (pfile, cur);
 	  ok = false;
-	  if (_cpp_save_parameter (pfile, macro, lex_identifier (pfile, cur)))
+	  if (_cpp_save_parameter (pfile, macro, id, id))
 	    break;
 	  cur = skip_whitespace (pfile, CUR (pfile->context),
 				 true /* skip_comments */);