I have now enabled a 64MB swap file, and this appears to improve the situation.
I still observe cyclical drops (reclamation?) and growth of buffer memory,
sometimes to very low levels (< 500KB), but the system no longer crashes
altogether nor becomes unreachable, presumably because availability of swap
somehow prevents buffers from running all the way to zero.
Anyone have any experience with modifying bdflush on an ext3 filesystem? I've
been wondering whether the patterns I've observed with growth and shrinkage of
buffers may be due to linux memory management functionality via some process
like bdflush.
Have been reading about bdflush and experimented with some new settings in
/proc/sys/vm/bdflush, but haven't been able to detect any impact or change in
behavior as as result of playing with bdflush variables. I'm running at ext3
filesystem, and have seen some posts suggesting bdflush may not work with ext3,
or that its use is unpredictable. Maybe this is why I see no differences.
--- In "tedapt" <> wrote:
>
> Thanks Breton. There is a lot of mystery in the Java garbage collection
> aspect of this problem. However, this is a large app and about two years in
> development, conversion to C is unfortunately not an option. What I really
> need is a better way to measure where the memory problem arises (or perhaps
> that is just a shadow of the real problem).
>
> I suppose at this point I'm really looking for more insight into what
> "Buffers" really means? I've read that this is either related to data blocks
> or interprocess communications. Hoping to better understand what that means
> (and ideally to somehow connect that to a particular thread) to perhaps nail
> down the few lines of Java code somewhere in the vast sea that could be
> causing the problem and rewrite that.
>
> Otherwise, hoping to identify some OS configuration variables (e.g., through
> sysctl) that might make a difference if tweaked.
>
> Have also tried allocating more and less heap to Java, with no effect, as
> well have tried touching large chunks of memory when my app initializes to
> try to ensure that enough memory is being allocated. But once again, the
> problem seems most closely tied to the "Buffers" value, not overall memory
> allocation.
>
> Thanks again for your recommendations.
>
> --- In "Breton M. Saunders" <breton.saunders@> wrote:
> >
> > Hi Tedapt,
> >
> > Looks like a kernel bug to me; most likely something with the archaic
> > version of linux that TS supplies with their 7xxx series of boards. Can
> > you record what is coming off of the console when this happens?
> >
> > Your best bet is to work around the problem by rewriting (or just
> > recompiling) your application in C.
> >
> > Your memory pattern demonstrates the typical java sawtooth graph -
> > after a collection memory is restored and the system continues
> > (presumably after an enormous pause). I would strongly recommend
> > rewriting (or just recompiling) your code in C and thinking quite hard
> > about the memory management of it. Remember you're running Java on an
> > embedded platform; thus the interpreter wastes memory, and power; and
> > with Java's garbage collection you'll never get real-time guarantees.
> >
> > I do a lot of work at a Java house that has about 1000 cores dedicated
> > for running servers. I (we) consistently watch memory management
> > problems bring down our projects after observing hours of sawtooth
> > graphs, like yours.
> >
> > -bms
> >
> >
> > tedapt wrote:
> > >
> > >
> > > I have posted the following Memory graphs
> > > <http://groups.yahoo.com/group/ts-7000/photos/album/917845361/pic/list?mode=tn&order=ordinal&start=1&dir=asc>
> > >
> > > which illustrate the problematic memory pattern and also the pattern
> > > observed when the application runs without any problems.
> > >
> > > --- In "tedapt" <tedapt@> wrote:
> > > >
> > > > Hi all -
> > > >
> > > > I have a data logging JamVM application on ts-7260 boards running
> > > Debian from an SD card. In some of my applications configurations, I
> > > find that the Buffers value obtained from `free` rises, then falls to
> > > zero over the course of about six hours. Once Buffers hits zero, the
> > > boards become unreachable, and all application processes appear to
> > > hang until a power cycle.
> > > >
> > > > In other configurations, the Buffers value rises and stays above the
> > > Free value, following a sawtooth pattern. On these boards, all remains
> > > OK for long periods. Occasionally I've seen these boards suddently
> > > exhibit a similar memory pattern, with Buffers suddenly declining
> > > until reaching zero, and again these boards become unreachable.
> > > >
> > > > I'm at a loss to understand what is happening. Have experimented
> > > with swap files, new versions of JamVM, code modifications and some
> > > sysctl settings, examined netstat, lsof output, all without any
> > > improvements.
> > > >
> > > > Has anyone every seen anything like this, or have any suggestions
> > > for solving this problem?
> > > >
> > > > Thanks!
> > > >
> > >
> > >
> > >
> >
>
------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/ts-7000/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/ts-7000/join
(Yahoo! ID required)
<*> To change settings via email:
<*> To unsubscribe from this group, send an email to:
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|