Project

General

Profile

Bug #5324

dcd_mgr buffer too small -- xxx Lost data.. messages

Added by Derik Barseghian about 8 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Normal
Category:
sensor-view
Target version:
Start date:
02/24/2011
Due date:
% Done:

0%

Estimated time:
Bugzilla-Id:
5324

Description

After running dcd_mgr for awhile I start getting messages like:
xxx Lost data: 2011-02-24T16:39:45.667Z CR800_sq311_1 3.3662283421E-01

Looking at SPAN's data_storage.c, this looks like intended behavior, at a certain point old data is discarded in ring buffer fashion. Data is discarded when MAX_DATA_QUEUE_SIZE is reached, and it seems hardcoded to 256.

This seems problematically small, and should probably be user configurable.

Sampling 3 sensors at 1sps, gives you 85 seconds before old data starts expiring. And I think if the spanToDT process goes down, our current flow will lose this data.

History

#1 Updated by Derik Barseghian about 8 years ago

Adding storage_interval=X to the cfg file enables writing files to gumstix disk. Data files are written as csv to spanddata/pending, and then sent out when a connection to the data port (55056) is made.

Dan's fixed 2 bugs wrt pending files:
1) empty files could be written to pending, and an error would occur when trying to send these.
2) files were not sent out in chronological order.

TODO: spanToDT's been using the live data port (55058); we'll have to change to use 55056.

#2 Updated by Daniel Crawl about 8 years ago

spanToDT now uses 55056.

Before closing this bug, we need to test what happens if the buffer fills faster than the storage interval.

#3 Updated by Daniel Crawl about 8 years ago

If there's no connection on the data port (55056), the buffer is written to storage every 5 seconds independent of the storage interval.

#4 Updated by Redmine Admin about 6 years ago

Original Bugzilla ID was 5324

Also available in: Atom PDF