Thanks Breton. There is a lot of mystery in the Java garbage collection aspect
of this problem. However, this is a large app and about two years in
development, conversion to C is unfortunately not an option. What I really need
is a better way to measure where the memory problem arises (or perhaps that is
just a shadow of the real problem).
I suppose at this point I'm really looking for more insight into what "Buffers"
really means? I've read that this is either related to data blocks or
interprocess communications. Hoping to better understand what that means (and
ideally to somehow connect that to a particular thread) to perhaps nail down
the few lines of Java code somewhere in the vast sea that could be causing the
problem and rewrite that.
Otherwise, hoping to identify some OS configuration variables (e.g., through
sysctl) that might make a difference if tweaked.
Have also tried allocating more and less heap to Java, with no effect, as well
have tried touching large chunks of memory when my app initializes to try to
ensure that enough memory is being allocated. But once again, the problem seems
most closely tied to the "Buffers" value, not overall memory allocation.
Thanks again for your recommendations.
--- In "Breton M. Saunders" <>
wrote:
>
> Hi Tedapt,
>
> Looks like a kernel bug to me; most likely something with the archaic
> version of linux that TS supplies with their 7xxx series of boards. Can
> you record what is coming off of the console when this happens?
>
> Your best bet is to work around the problem by rewriting (or just
> recompiling) your application in C.
>
> Your memory pattern demonstrates the typical java sawtooth graph -
> after a collection memory is restored and the system continues
> (presumably after an enormous pause). I would strongly recommend
> rewriting (or just recompiling) your code in C and thinking quite hard
> about the memory management of it. Remember you're running Java on an
> embedded platform; thus the interpreter wastes memory, and power; and
> with Java's garbage collection you'll never get real-time guarantees.
>
> I do a lot of work at a Java house that has about 1000 cores dedicated
> for running servers. I (we) consistently watch memory management
> problems bring down our projects after observing hours of sawtooth
> graphs, like yours.
>
> -bms
>
>
> tedapt wrote:
> >
> >
> > I have posted the following Memory graphs
> > <http://groups.yahoo.com/group/ts-7000/photos/album/917845361/pic/list?mode=tn&order=ordinal&start=1&dir=asc>
> >
> > which illustrate the problematic memory pattern and also the pattern
> > observed when the application runs without any problems.
> >
> > --- In "tedapt" <tedapt@> wrote:
> > >
> > > Hi all -
> > >
> > > I have a data logging JamVM application on ts-7260 boards running
> > Debian from an SD card. In some of my applications configurations, I
> > find that the Buffers value obtained from `free` rises, then falls to
> > zero over the course of about six hours. Once Buffers hits zero, the
> > boards become unreachable, and all application processes appear to
> > hang until a power cycle.
> > >
> > > In other configurations, the Buffers value rises and stays above the
> > Free value, following a sawtooth pattern. On these boards, all remains
> > OK for long periods. Occasionally I've seen these boards suddently
> > exhibit a similar memory pattern, with Buffers suddenly declining
> > until reaching zero, and again these boards become unreachable.
> > >
> > > I'm at a loss to understand what is happening. Have experimented
> > with swap files, new versions of JamVM, code modifications and some
> > sysctl settings, examined netstat, lsof output, all without any
> > improvements.
> > >
> > > Has anyone every seen anything like this, or have any suggestions
> > for solving this problem?
> > >
> > > Thanks!
> > >
> >
> >
> >
>
------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/ts-7000/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/ts-7000/join
(Yahoo! ID required)
<*> To change settings via email:
<*> To unsubscribe from this group, send an email to:
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|