Thank you, Paul! I have started a new round of testing that includes periodic
sampling of output from the 'free' command and the SUnreclaim line from
/proc/meminfo.
The test I just finished showed that Linux grabs all but the last ~1.3 MB of
available memory at which point it frees about 1.1 MB and continues running,
using memory in blocks of 12 KB. This just repeats over and over. When our
application failed, there was still plenty of memory left.
So, memory leaks do not appear to be causing our problem. Nor does an errant
pointer - if the application had strayed out of bounds, we would have seen a
segment violation message. In fact, we see no messages at all, a sign the
application was killed via a signal from the kernel.
I tried compiling with debug information and starting the application via GDB,
but the system was so overloaded that it failed within seconds of starting.
The sbuslock() assert("r == 0") call was made, showing a semop(SEM_UNDO)
failure. Without GDB, the assert is never seen, so my next test will be
without it.
I will notify the group of any other findings I stumble upon. ;-)
Regards, Mitch
--- In "ptreos2" <> wrote:
>
>
> We see this same problem on the same HW and linux kernel. The only network
> activity we have is for a web server. If no browser points to the web server
> we stay up forever. Put a browser on it which continuously updates pages we
> run out of free memory within 20 hours.
>
> Look at /proc/meminfo and you will see SUnreclaimable steadily increasing.
> This is the kernel leaking memory. However it doesn't happen all the time. It
> seems one must first see an nbd error in syslog. The error I see looks like
> this:
>
> Jul 17 14:22:14 ts7500 kernel: [ 6061.930000] nbd1: Other side returned error
> (1)
> Jul 17 14:22:14 ts7500 kernel: [ 6061.930000] end_request: I/O error, dev
> nbd1, sector 1073741696
> Jul 17 14:22:14 ts7500 kernel: [ 6061.930000] Buffer I/O error on device
> nbd1, logical block 134217712
>
> Only after this error occurs do I then see the kernel memory leak.
> I am talking to a tech rep about this issue but is going to take some time to
> resolve. We may need to go to a new kernel although I would much rather have
> a field deployable fix.
>
> Paul
>
>
> --- In "al" <mitch.stanek@> wrote:
> >
> > We are using a TS-7500 for a network-intensive application that sends and
> > receives multiple messages (TCP) per second. The application does no
> > run-time memory allocation; all allocation is done at power-up. The
> > application is supposed to run continuously for months, but our tests show
> > it stops running after about 4 days and has a memory leakage of about 2
> > MB/day! The TS-7500 has Linux 2.6.24.4.
> >
> > When the application starts, 'free' reports 1.4 MB free + 15 MB
> > cache/buffer. Just before the application is killed by Linux (4.5 days
> > later), the free memory is 1.3 MB + 7.2 MB cache/buffer.
> >
> > It appears that the Linux network stack has some serious memory leaks.
> >
> > We are using the fast-boot option, running the Busybox Linux that comes in
> > the Flash memory. We tried the slow-boot option using the Linux 2.6.24
> > kernel, but got the same results, so it seems endemic to the Linux network
> > stack. To ensure the application is not leaking memory, we will run
> > Valgrind.
> >
> > In the mean time, has anyone in this group encountered this memory leakage
> > in network-intensive applications? If so, were you able to fix it?
> >
> > Kind regards,
> > Mitch
> >
>
------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/ts-7000/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/ts-7000/join
(Yahoo! ID required)
<*> To change settings via email:
<*> To unsubscribe from this group, send an email to:
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|