Bug #5324
closeddcd_mgr buffer too small -- xxx Lost data.. messages
0%
Description
After running dcd_mgr for awhile I start getting messages like:
xxx Lost data: 2011-02-24T16:39:45.667Z CR800_sq311_1 3.3662283421E-01
Looking at SPAN's data_storage.c, this looks like intended behavior, at a certain point old data is discarded in ring buffer fashion. Data is discarded when MAX_DATA_QUEUE_SIZE is reached, and it seems hardcoded to 256.
This seems problematically small, and should probably be user configurable.
Sampling 3 sensors at 1sps, gives you 85 seconds before old data starts expiring. And I think if the spanToDT process goes down, our current flow will lose this data.
Updated by Derik Barseghian about 13 years ago
Adding storage_interval=X to the cfg file enables writing files to gumstix disk. Data files are written as csv to spanddata/pending, and then sent out when a connection to the data port (55056) is made.
Dan's fixed 2 bugs wrt pending files:
1) empty files could be written to pending, and an error would occur when trying to send these.
2) files were not sent out in chronological order.
TODO: spanToDT's been using the live data port (55058); we'll have to change to use 55056.
Updated by Daniel Crawl about 13 years ago
spanToDT now uses 55056.
Before closing this bug, we need to test what happens if the buffer fills faster than the storage interval.
Updated by Daniel Crawl about 13 years ago
If there's no connection on the data port (55056), the buffer is written to storage every 5 seconds independent of the storage interval.