dcd_mgr buffer too small -- xxx Lost data.. messages
After running dcd_mgr for awhile I start getting messages like:
xxx Lost data: 2011-02-24T16:39:45.667Z CR800_sq311_1 3.3662283421E-01
Looking at SPAN's data_storage.c, this looks like intended behavior, at a certain point old data is discarded in ring buffer fashion. Data is discarded when MAX_DATA_QUEUE_SIZE is reached, and it seems hardcoded to 256.
This seems problematically small, and should probably be user configurable.
Sampling 3 sensors at 1sps, gives you 85 seconds before old data starts expiring. And I think if the spanToDT process goes down, our current flow will lose this data.
Updated by Derik Barseghian almost 12 years ago
Adding storage_interval=X to the cfg file enables writing files to gumstix disk. Data files are written as csv to spanddata/pending, and then sent out when a connection to the data port (55056) is made.
Dan's fixed 2 bugs wrt pending files:
1) empty files could be written to pending, and an error would occur when trying to send these.
2) files were not sent out in chronological order.
TODO: spanToDT's been using the live data port (55058); we'll have to change to use 55056.