[openssl.org #1169] Crash in OpenSSL - write_pending

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[openssl.org #1169] Crash in OpenSSL - write_pending

Rich Salz via RT

Hi,

I have discovered something that seems to be a crash in OpenSSL running with
Apache+mod_ssl. It happens when SSL connection times out (I simulate it by
suspending the reader process, but it happens also in production with very slow
clients). Apache gives message: send mmap timed out and then segfault happens.
The backtrace is as follows, on OpenSSL 0.9.8+mod_ssl 2.8.23+Apache 1.3.33:

#4  <signal handler called>
#5  0x40271769 in write_pending (s=0x85160d0,
     buf=0x43bf7000 "se : + \"= HM_f_EvalParameters(\""..., len=32768)
     at s2_pkt.c:501
#6  0x40271bc4 in do_ssl_write (s=0x85160d0,
     buf=0x43bf7000 "se : + \"= HM_f_EvalParameters(\""..., len=32744)
     at s2_pkt.c:647
#7  0x40271609 in ssl2_write (s=0x85160d0, _buf=0x43bf7000, len=32768) at
s2_pkt.c:450
#8  0x402903ae in SSL_write (s=0x85160d0, buf=0x43bf7000, num=32768) at
ssl_lib.c:894
#9  0x0822ac23 in ssl_io_hook_write (fb=0x8339a5c,
     buf=0x43bf7000 "se : + \"= HM_f_EvalParameters(\""..., len=32768)
     at ssl_engine_io.c:385
#10 0x08265497 in ap_hook_call_func (ap=0xbffff6b4, he=0x831f6b0, hf=0x8322910)
at ap_hook.c:649
#11 0x08264c25 in ap_hook_call (hook=0x82b89af "ap::buff::write") at
ap_hook.c:382
#12 0x0823f0d4 in ap_write (fb=0x8339a5c, buf=0x43bf7000, nbyte=32768) at
buff.c:318
#13 0x08240b57 in buff_write (fb=0x8339a5c, buf=0x43bf7000, nbyte=32768) at
buff.c:365
#14 0x0823ffe1 in write_with_errors (fb=0x8339a5c, buf=0x43bf7000, nbyte=32768)
at buff.c:1133
#15 0x082400a5 in bcwrite (fb=0x8339a5c, buf=0x43bf7000, nbyte=32768) at
buff.c:1170
#16 0x0824054d in ap_bwrite (fb=0x8339a5c, buf=0x43bf7000, nbyte=32768) at
buff.c:1384
#17 0x082541fa in ap_send_mmap (mm=0x43bd7000, r=0x83fe9d4, offset=131072,
length=316620) at http_protocol.c:2571
#18 0x08249947 in default_handler (r=0x83fe9d4) at http_core.c:4227
#19 0x082415d6 in ap_invoke_handler (r=0x83fe9d4) at http_config.c:487
#20 0x082575db in process_request_internal (r=0x83fe9d4) at http_request.c:1298
#21 0x0825763c in ap_process_request (r=0x83fe9d4) at http_request.c:1314
#22 0x0824e1c9 in child_main (child_num_arg=64) at http_main.c:4872
#23 0x0824e45b in make_child (s=0x8316c6c, slot=64, now=1121006025) at
http_main.c:5051
#24 0x0824e50c in startup_children (number_to_start=6) at http_main.c:5078
#25 0x0824ec1a in standalone_main (argc=1, argv=0xbffffab4) at http_main.c:5410
#26 0x0824f4ab in main (argc=1, argv=0xbffffab4) at http_main.c:5767

(first 3 frames are the custom SIGSEGV handler). Looking at s2_pkt.c line 501 I
see:

                 if (i == s->s2->wpend_len)
                         {

and I can see in the debugger that s->s2 is 0 there, so this seems to be the
reason for the crash. I'm not sure if the OpenSSL is the reason for the crash
or Apache is using it in a wrong way, but the SEGV seems to be happening in
OpenSSL code so I send it to this list, if it belongs elsewhere please point me
to the right place.

I also tried to research it somewhat more, it seems that what is happening
is that on some stage when timeout happens OpenSSL returns OK result from
SSL_write but somehow s->s2 and s->s3 become NULL, so on next SSL_write it
crashes. I'd appreciate guidance on what could be the problem.

OS is Linux on x86, OpenSSL compiled by gcc 2.96.

TIA,
--
Stanislav Malyshev, Zend Products Engineer
[hidden email]  http://www.zend.com/ +972-3-6139665 ext.115

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #1169] Crash in OpenSSL - write_pending

Dan Kegel-2
Stanislav Malyshev via RT wrote:

> I have discovered something that seems to be a crash in OpenSSL running with
> Apache+mod_ssl. It happens when SSL connection times out (I simulate it by
> suspending the reader process, but it happens also in production with very slow
> clients). Apache gives message: send mmap timed out and then segfault happens.
> ...
> I'm not sure if the OpenSSL is the reason for the crash
> or Apache is using it in a wrong way, but the SEGV seems to be happening in
> OpenSSL code so I send it to this list, if it belongs elsewhere please point me
> to the right place.
>
> I also tried to research it somewhat more, it seems that what is happening
> is that on some stage when timeout happens OpenSSL returns OK result from
> SSL_write but somehow s->s2 and s->s3 become NULL, so on next SSL_write it
> crashes. I'd appreciate guidance on what could be the problem.
>
> OS is Linux on x86, OpenSSL compiled by gcc 2.96.

You might want to try reproducing the problem under Valgrind;
it might give you earlier information about the
error, and possibly help you find out whether it's
Apache or OpenSSL at fault.
- Dan

--
Trying to get a job as a c++ developer?  See http://kegel.com/academy/getting-hired.html
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]