Project

General

Profile

Bug #5324

dcd_mgr buffer too small -- xxx Lost data.. messages

Added by Derik Barseghian over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Normal
Category:
sensor-view
Target version:
Start date:
02/24/2011
Due date:
% Done:

0%

Estimated time:
Bugzilla-Id:
5324

Description

After running dcd_mgr for awhile I start getting messages like:
xxx Lost data: 2011-02-24T16:39:45.667Z CR800_sq311_1 3.3662283421E-01

Looking at SPAN's data_storage.c, this looks like intended behavior, at a certain point old data is discarded in ring buffer fashion. Data is discarded when MAX_DATA_QUEUE_SIZE is reached, and it seems hardcoded to 256.

This seems problematically small, and should probably be user configurable.

Sampling 3 sensors at 1sps, gives you 85 seconds before old data starts expiring. And I think if the spanToDT process goes down, our current flow will lose this data.

History

#1 Updated by Derik Barseghian over 10 years ago

Adding storage_interval=X to the cfg file enables writing files to gumstix disk. Data files are written as csv to spanddata/pending, and then sent out when a connection to the data port (55056) is made.

Dan's fixed 2 bugs wrt pending files:
1) empty files could be written to pending, and an error would occur when trying to send these.
2) files were not sent out in chronological order.

TODO: spanToDT's been using the live data port (55058); we'll have to change to use 55056.

#2 Updated by Daniel Crawl over 10 years ago

spanToDT now uses 55056.

Before closing this bug, we need to test what happens if the buffer fills faster than the storage interval.

#3 Updated by Daniel Crawl over 10 years ago

If there's no connection on the data port (55056), the buffer is written to storage every 5 seconds independent of the storage interval.

#4 Updated by Redmine Admin over 8 years ago

Original Bugzilla ID was 5324

Also available in: Atom PDF