Re: [ts-7000] Re: syncing time on many TS-7550

[Jonathan Klay - NOAA Federal <>]

Sounds very similar to our operation. We have 1 channel at 40Hz, 3 at 1 Hz.  I was not buffering, so getting about 40 small (55 to 70 byte) packets every second.  I think slow data rate makes a difference.  The TCP window stays small, and when packet lost, will not do SACK because window smaller than 4 packets, so must timeout before retransmit.  Always 27 seconds minimum though, no matter what I do, but that's better than 51 seconds, or several minutes.  I think the Roundtrip Time Out value is compiled into the kernel.

Maybe I should buffer more.  I started buffering for 1 second Friday, which is oddly worse, even though every second or third packet is full-size. Lost packet means 27 second to retransmit, then packet before retransmitted always lost, 27 seconds to retransmit that, then back to normal operation.  I was going to set it back and just live with the occasional delays.

Perhaps your 5 send gaps are retransmit from lost packet.  Also, I'm using the linuxrc-sdroot-readonly boot, I've begun wondering if that affects TCP, but time-consuming to test.

Jonathan

On Sun, Dec 2, 2012 at 5:15 AM, bru22003 <> wrote:

Hi Jonathan of Ocean-research

I have been off line for a while.

While we can not claim things solved they are working much better for us.  At the remote sites digitized 4 channels at 50 hz each and send then over the serial port to the 7550 and export them via TCP.  We buffer abut 8 second.  At the central we get all of the data from the remotes synced in time.  We still have several gaps a day at the central mostly about 5 seconds wide. Since we are looking at the data as a network and the gaps are asynchronous we have deemed thing acceptable and are waiting for TS to come our with a new board with the same foot print as the 7550.

We have forced the 7550 to 10 instead of 10/100 and that seemed to help but we ha not gone as far as you have in the config files.

Also we are no longer sending data through the vtun, we only use the tunnel for control.

If you think we can help let's take the thread offline.

Angel

--- In "ocean_research_jk" <> wrote:
>
> I've been adjusting tcp parameters for a week, capturing packets, and laboriously looking through them.  By increasing the tcp memory parameters (which are by default quite low for a linux system) and changing from reno to cubic for congestion control, I've gotten it to "only" take 26 to 51 seconds, vs 4 to 10 minutes, to retransmit a lost segment.  If this is not just because I screwed something up (possible!), or a bizarre firewall issue, but a feature of the kernel's tcp stack, it will play havoc with you trying to synchronize data packets if you have any loss.
> Jonathan
>

> --- In "angel@" <angel@> wrote:
> >
> > On 11/15/2012 07:08 PM, ocean_research_jk wrote:
> > >
> > > I used chrony, which was small and easy to install.
> > >
> > Hi All,
> >
> > I have not reported back to the group because we are still doing a bunch
> > of testing. I think we have the synchrony thing mostly solved but are
> > still working on the tunnel. We are using vtun and now trying out
> > openvpn. The 75XX has real troubles with the network, we have forced the
> > 75xx to 10 (not 10/100) and still have real troubles with some of the cards.
> >
> > When we get thing working well I will report back.
> >
> > Angel
> >
> >
> >
> >
> > >
> > > --- In <mailto:ts-7000%2540yahoogroups.com>,
> > > Joseph Bouchard <jbouchard@> wrote:
> > > >
> > > > Debian's ntpdate package is simple and can probably do what you want. I
> > > > haven't tried to use it to that degree of accuracy, but I suspect if
> > > you
> > > > set the sync time fairly tight (like maybe every 5 minutes) the drift
> > > > over that period will be a few milliseconds. I would try it, if it
> > > > doesn't work, then look for a more complex solution.
> > > >
> > > > On 10/24/2012 07:23 PM, Walter Marvin wrote:
> > > > >
> > > > >
> > > > > Unfortunately, this is not a valid test, which is typical of
> > > > > Technologic. First of all it is anedotal, and no real measurement is
> > > > > made. Second it is probably not run long enough to make a statically
> > > > > significant conclusion (although they don't say how long their "burn
> > > > > in" runs) Third it is probably run with a simplistic program which
> > > are
> > > > > all the same, and presents no load or variation on the system.
> > > > >
> > > > > you have been warned
> > > > >
> > > > > Walter
> > > > >
> > > > >
> > > > > ----------------------------------------------------------
> > > > > *From:* Angel <angel@>
> > > > > *To:* <mailto:ts-7000%2540yahoogroups.com>
> > > > > *Sent:* Wednesday, October 24, 2012 5:37 PM
> > > > > *Subject:* Re: [ts-7000] syncing time on many TS-7550
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Just as an update. I got this from Grant at TS.
> > > > >
> > > > > When TS-7550s are tested with the TS-752, we actually run a stress
> > > test
> > > > > that proves out the validity of a software solution. 20 or more units
> > > > > are placed on a burn-in rack and they automatically run a burn-in/test
> > > > > script. They synchronize their system time from our local NTP server,
> > > > > and then they begin toggling relays and LEDs at 1Hz. The goal of this
> > > > > is just to test the relays, but the ear and eye can tell that they are
> > > > > reasonably synchronized and all the relays click as one. The
> > > timing was
> > > > > implemented with the attached C program and a shell wrapper that
> > > uses it
> > > > > to wait for the right time to toggle things using ts7500ctl.
> > > > >
> > > > > We are coding some stuff now and will have some results in a few days,
> > > > > we need to butcher some harware. ntpd displines the TS clock to just a
> > > > > few milliseconds and I suspect that it what I am going to do will
> > > work.
> > > > > If anyone is interested in the code snippet that Grant sent just ask.
> > > > >
> > > > > regards,
> > > > >
> > > > > Angel
> > > > >
> > > > > On 10/24/2012 04:30 PM, Ken Lu wrote:
> > > > > >
> > > > > > I did on TS-7400(connect to laptop by Ethernet), modified a telnet
> > > > > > program, to synchronize PC's time, which comes from GPS connected to
> > > > > > PC. But found that TS-Linux's date command can only set to the
> > > > > > minute, so need to wait within a minute, depends on start
> > > setting time.
> > > > > >
> > > > > > --- On *Wed, 10/24/12, Walter Marvin /<walter.marvin@
> > > > > <mailto:walter.marvin%2540ymail.com>>/* wrote:
> > > > > >
> > > > > >
> > > > > > From: Walter Marvin <walter.marvin@
> > > > > <mailto:walter.marvin%2540ymail.com>>
> > > > > > Subject: Re: [ts-7000] syncing time on many TS-7550
> > > > > > To: " <mailto:ts-7000%2540yahoogroups.com>
> > > <mailto:ts-7000%2540yahoogroups.com>"
> > > > > < <mailto:ts-7000%2540yahoogroups.com>
> > > <mailto:ts-7000%2540yahoogroups.com>>
> > > > > > Received: Wednesday, October 24, 2012, 10:36 AM
> > > > > >
> > > > > > You could make a real time clock driver and use the hardware's
> > > > > > alarm feature. this may mean using a different clock chip .
> > > > > > Fortunately there are a number of IIC clock chips that can do this.
> > > > > >
> > > > > >
> > > > > > ----------------------------------------------------------
> > > > > > *From:* Angel <angel@ <mailto:angel%2540tierrasaltas.net>>
> > > > > > *To:* <mailto:ts-7000%2540yahoogroups.com>
> > > <mailto:ts-7000%2540yahoogroups.com>
> > > > > > *Sent:* Wednesday, October 24, 2012 9:30 AM
> > > > > > *Subject:* [ts-7000] syncing time on many TS-7550
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I have an application where I need to sync many TS-7550 in the
> > > > > > field to
> > > > > > within one tenth of a second of each other. I know I could do this
> > > > > > with
> > > > > > a GPS and a PPS on each one but the cost and trouble to place the
> > > > > > antenna makes that a non-solution.
> > > > > >
> > > > > > We have a hardware one pulse per second on the daughter card that is
> > > > > > deployed with system so we can generate one second data packages and
> > > > > > send them over TCP/IP. All the 7550 have a RTC and all have ntpd
> > > > > > running so I can discipline the hardware clocks to better than the
> > > > > > 1/10th second requirement.
> > > > > >
> > > > > > What I am at a loss on how to do is to sync the one second data
> > > > > > packets
> > > > > > to the change of second of the RTC or any other stable time source.
> > > > > >
> > > > > > TS says that the FPGA can be asked to make a one pulse per
> > > second but
> > > > > > that syncing the PPS pulse to the RTC is not sure.
> > > > > >
> > > > > > If anyone has an idea on how to do this I would love to hear
> > > from you.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Angel
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>




------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/ts-7000/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/ts-7000/join
    (Yahoo! ID required)

<*> To change settings via email:
   
   

<*> To unsubscribe from this group, send an email to:
   

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

--
Jonathan Klay
System Administrator -- System Manager
PMEL CNSD -- Vents Program

NOAA Using Google Get To Know Google Apps, Learn About UMS

__._,_.___

Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: =Email Delivery: Digest | m("yahoogroups.com?subject","ts-7000-fullfeatured");=Change Delivery Format: Fully Featured">Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | =Unsubscribe

__,_._,___