Prevent tail call optimization with _gcry_burn_stack

* configure.ac: New check, HAVE_GCC_ASM_VOLATILE_MEMORY.
* src/g10lib.h (_gcry_burn_stack): Rename to __gcry_burn_stack.
(__gcry_burn_stack_dummy): New.
(_gcry_burn_stack): New macro.
* src/misc.c (_gcry_burn_stack): Rename to __gcry_burn_stack.
(__gcry_burn_stack_dummy): New.
--

Tail call optimization can turn _gcry_burn_stack call in to tail jump. When
this happens, stack pointer is restored to initial state of current function.
This causes problem for _gcry_burn_stack because its callers do not count in
current function stack depth.

One solution is to prevent gcry_burn_stack being tail optimized by inserting
dummy function call behind it. Another would be to add memory barrier 'asm
volatile("":::"memory")' behind every _gcry_burn_stack call. This however
requires GCC asm support from compiler.

Patch adds detection for memory barrier support and when available uses
memory barrier to prevent when tail call optimization. If not available
dummy function call is used instead.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>

1 files changed, 8 insertions, 1 deletions

diff --git a/src/misc.c b/src/misc.c
index 912039a5..9b30ac3d 100644
--- a/src/misc.c
+++ b/src/misc.c
@@ -438,7 +438,7 @@ _gcry_log_printsxp (const char *text, gcry_sexp_t sexp)
 
 
 void
-_gcry_burn_stack (unsigned int bytes)
+__gcry_burn_stack (unsigned int bytes)
 {
 #ifdef HAVE_VLA
     /* (bytes == 0 ? 1 : bytes) == (!bytes + bytes) */
@@ -456,6 +456,13 @@ _gcry_burn_stack (unsigned int bytes)
 #endif
 }
 
+#ifndef HAVE_GCC_ASM_VOLATILE_MEMORY
+void
+__gcry_burn_stack_dummy (void)
+{
+}
+#endif
+
 void
 _gcry_divide_by_zero (void)
 {