Bug #5324

dcd_mgr buffer too small -- xxx Lost data.. messages

Added by Derik Barseghian about 11 years ago. Updated about 11 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


After running dcd_mgr for awhile I start getting messages like:
xxx Lost data: 2011-02-24T16:39:45.667Z CR800_sq311_1 3.3662283421E-01

Looking at SPAN's data_storage.c, this looks like intended behavior, at a certain point old data is discarded in ring buffer fashion. Data is discarded when MAX_DATA_QUEUE_SIZE is reached, and it seems hardcoded to 256.

This seems problematically small, and should probably be user configurable.

Sampling 3 sensors at 1sps, gives you 85 seconds before old data starts expiring. And I think if the spanToDT process goes down, our current flow will lose this data.


#1 Updated by Derik Barseghian about 11 years ago

Adding storage_interval=X to the cfg file enables writing files to gumstix disk. Data files are written as csv to spanddata/pending, and then sent out when a connection to the data port (55056) is made.

Dan's fixed 2 bugs wrt pending files:
1) empty files could be written to pending, and an error would occur when trying to send these.
2) files were not sent out in chronological order.

TODO: spanToDT's been using the live data port (55058); we'll have to change to use 55056.

#2 Updated by Daniel Crawl about 11 years ago

spanToDT now uses 55056.

Before closing this bug, we need to test what happens if the buffer fills faster than the storage interval.

#3 Updated by Daniel Crawl about 11 years ago

If there's no connection on the data port (55056), the buffer is written to storage every 5 seconds independent of the storage interval.

#4 Updated by Redmine Admin about 9 years ago

Original Bugzilla ID was 5324

Also available in: Atom PDF