VegBank: Issues
https://projects.ecoinformatics.org/ecoinfo/
https://projects.ecoinformatics.org/ecoinfo/ecoinfo/favicon.ico?1469134036
2006-09-07T13:55:35Z
Ecoinformatics Redmine
Redmine
Bug #2540 (Resolved): disk Cache full?
https://projects.ecoinformatics.org/ecoinfo/issues/2540
2006-09-07T13:55:35Z
Michael Lee
mlee@nceas.ucsb.edu
<p>There is a message in the log on VegBank that the disk cache is full:</p>
<p>WARN datasource.DBModelBeanReader: Disk Cache Full: 10000 files.<br />DEBUG datasource.DBModelBeanReader: Deleted from Disk Cache: VB.ob.26759.CHAT59<br />DEBUG datasource.DBModelBeanReader: Added to Disk Cache: VB.PC.6832.ABRONIAMICRA</p>
<p>do we just need to up the MAX_DISK_CACHE_SIZE<br />the odd thing is, the disk cache has many more files in it that 10000: <br />vegbank=# select count(1) from dba_xmlcache;<br /> count <br />--------<br /> 107552<br />(1 row)</p>
Bug #2536 (Resolved): XML download files must validate
https://projects.ecoinformatics.org/ecoinfo/issues/2536
2006-09-05T20:18:52Z
Michael Lee
mlee@nceas.ucsb.edu
<p>dateTime format was fixed, but nulls still throw it for a loop- the field should either be omitted, or we could redefine our schema to allow either nulls or xs:DateTime (I could do the latter)</p>
<p>The UTF-8 encoding must be adhered to. In the worst case scenario, <br /><plantName.plantName>Athyrium filix-femina (L.) Roth ssp. asplenioides (Michx.) Hultén</plantName.plantName></p>
<p>got rendered as:<br /><plantName.plantName>Athyrium filix-femina (L.) Roth ssp. asplenioides (Michx.) Hult?/plantName.plantName><br />in the XML (with the ? being an unprintable character). This wiped out hte closing tag for plantName.plantName and made the file not valid</p>
Bug #2526 (Resolved): Denorms run 26 times on XML Loading
https://projects.ecoinformatics.org/ecoinfo/issues/2526
2006-08-26T02:32:07Z
Michael Lee
mlee@nceas.ucsb.edu
<p>When I load a simple XML file to Aldo, the denorms run through the complete cycle of denorms, in order, but they run 26 times. So I think the process is getting kicked off for each table or something.</p>
<p>testAdded105Fields_newFields_withPrCMSM.xml<br />is the simple XML that I loaded and this is on //aldo/tmp and also in //aldo/home/lee</p>
Bug #2522 (Resolved): XML comments should be turned off
https://projects.ecoinformatics.org/ecoinfo/issues/2522
2006-08-25T01:52:12Z
Michael Lee
mlee@nceas.ucsb.edu
<p>currently the XML is created with a lot of debugging statements inserted. While these aren't horrible, it would be a lot better without them, both in terms of space used, professionalism, and speed of download.</p>
<p>for example:<br /> <coverIndex.COVERINDEX_ID>8</coverIndex.COVERINDEX_ID><br /> <!-- Ignoring coverIndex.COVERMETHOD_ID --><br /> <coverIndex.coverCode>8</coverIndex.coverCode></p>
<p>basically all the comments in that document can be turned off.</p>
Bug #2521 (Resolved): XML needs to reference new xsd
https://projects.ecoinformatics.org/ecoinfo/issues/2521
2006-08-25T01:46:11Z
Michael Lee
mlee@nceas.ucsb.edu
<p>At the top of our XML documents, we have a declaration:<br /><VegBankPackage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://vegbank.org/vegdocs/xml/vegbank-data-ver1.0.2usage.xsd"></p>
<p>but this references an old xsd version. This is a bit confusing, as we have different version of the model and different versions of the xsd.</p>
<p>It should reference:<br /><a class="external" href="http://vegbank.org/vegdocs/xml/vegbank-data-ver1.0.5-full.xsd">http://vegbank.org/vegdocs/xml/vegbank-data-ver1.0.5-full.xsd</a><br />(working on creating that!)</p>
Bug #2520 (Resolved): XML Doesn't validate: DateTime invalid
https://projects.ecoinformatics.org/ecoinfo/issues/2520
2006-08-25T01:44:24Z
Michael Lee
mlee@nceas.ucsb.edu
<p>In our XML stream, we have tons of date fields that are now being output like so:</p>
<p><observation.obsStartDate>2001-10-24 01:00:00-06</observation.obsStartDate></p>
<p>but we need to make them XML conformant, i.e. like this:<br /><observation.obsStartDate>2001-10-24T01:00:00-06:00</observation.obsStartDate></p>
<p>which matches the pattern:<br />\-?([1-9][0-9][0-9][0-9]+)|(0[0-9][0-9][0-9])\-(0[1-9])|(1[0-2])\-(0[1-9])([12][0-9])|(3<sup><a href="#fn01">01</a></sup>)<br /> T(([01][0-9])|(2[0-3]):[0-5][0-9]:([0-5][0-9])(\.[0-9]+)?)|(24:00:00(\.0+)?)<br /> ([+\-](0[0-9])|(1[0-4]):[0-5][0-9])?</p>
<p>as specified by w3.org: <br /><a class="external" href="http://www.w3.org/TR/xmlschema11-2/#dateTime">http://www.w3.org/TR/xmlschema11-2/#dateTime</a></p>
<p>I don't know if other fields fail to validate, it's unlikely that other are wrong as other data types are more compatible.</p>
Bug #2497 (Resolved): Denorm utility incorrect in order of statements (based on tables, not seque...
https://projects.ecoinformatics.org/ecoinfo/issues/2497
2006-07-20T22:22:48Z
Michael Lee
mlee@nceas.ucsb.edu
<p>I think the denorm utility updates after an XML load from table-to-table. This is incorrect. It should go in the order of the numbered dnrm- statements in SQLStore, NOT BY TABLE.</p>
<p>MLee should check that this is indeed happening and possibly suggest a solution in the java utility.</p>
Bug #2482 (Resolved): automate caching of data for homepage -- replace ant function
https://projects.ecoinformatics.org/ecoinfo/issues/2482
2006-06-28T17:21:19Z
Michael Lee
mlee@nceas.ucsb.edu
<p>We need to automate the caching of data that is displayed on the homepage with a cron job. Currently, the caching is done from the ant target "cache" in /vegbank/web/. We could either use a cron job to update the site every so often, or run this after someone loads more data to the database (after denormalization is complete). Or perhaps both. If we use the cron job, this bug will depend on bug 2478.</p>
<p>/vegbank/views/raw/* needs to be wget into /vegbank/cache/views/raw/*</p>
<p>Once this is done, we can remove the "cache" target in the /web/build.xml file (or just make it do nothing as removing the target could screw up dependencies).</p>
Bug #2481 (Resolved): Fix Backup Strategy: where backups go
https://projects.ecoinformatics.org/ecoinfo/issues/2481
2006-06-28T17:12:35Z
Michael Lee
mlee@nceas.ucsb.edu
<p>+<ins>+ This bug was initially created as a clone of Bug <a class="issue tracker-1 status-3 priority-5 priority-highest closed" title="Bug: Fix Backup Strategy -- delete extra backups from time to time (Resolved)" href="https://projects.ecoinformatics.org/ecoinfo/issues/2395">#2395</a> +</ins>+</p>
<p>We need to make sure that this backup is not just to VegBank- a copy needs to be distributed, either by email or network to an alternate machine.</p>
<p>Colby needs to do this, or at least help.<br />/usr/vegbank/backup is where the backups are stored on VegBank, and the latest copy of the database should be backed up to "elsewhere" from time to time. Not sure what that "time to time" really is.</p>
Bug #2478 (Resolved): Implement Nightly/Weekly Cron (SQL)
https://projects.ecoinformatics.org/ecoinfo/issues/2478
2006-06-28T16:00:38Z
Michael Lee
mlee@nceas.ucsb.edu
<p>The URL describes tasks needed for nightly, or perhaps weekly for VegBank. MLee should write the SQL that accomplishes these things, then Chad could implement the cron job.</p>
<ul>
<li>make automated caching of home page data + wget on all the cache data.</li>
<li>make backup of database (remove backups older than 4 months old too??)</li>
<li>plus we should probably vacuum analyze too each night</li>
<li>plut would be nice to run SQL each night: dba_current-table-stats.sql in src/sql - this gets stats on what's currently in the db in terms of record counts for each table.</li>
<li>delete anon datasets > 24 hours old</li>
<li>delete users datasets with stopDate more than one month ago</li>
</ul>
Bug #2444 (Resolved): user passwords are stored in clear text in the database
https://projects.ecoinformatics.org/ecoinfo/issues/2444
2006-05-25T18:58:59Z
Chad Berkley
berkley@nceas.ucsb.edu
<p>User's passwords stored in the usr.password table/column are stored in clear text. This allows a system administrator to view any users password. These should be encrypted so that no one can ever see another person's password.</p>
Bug #2435 (Resolved): Store XML file's AccessionCodes in userDatasetItem
https://projects.ecoinformatics.org/ecoinfo/issues/2435
2006-05-09T15:20:55Z
Michael Lee
mlee@nceas.ucsb.edu
<p>When XML is being loaded into the database, accessionCodes exist in the user's XML document that come from the user's VegBranch database. As new items are loaded into the database, corresponding records are added to the userDatasetItem table to indicate what was loaded in a particular XML file. There is a field on userDatasetItem called externalaccessioncode which is used to store accessioncodes from other databases (VegBranch, some other version of VegBank, perhaps some other GUID?). It needs to be populated, but isn't yet.</p>
Bug #2434 (Resolved): Profiling of XML Loader's Performance
https://projects.ecoinformatics.org/ecoinfo/issues/2434
2006-05-09T15:15:48Z
Michael Lee
mlee@nceas.ucsb.edu
<p>During the week of 5/8/2006, the XML Loader should be profiled to see why it takes as long as it does and whether there are places where we could improve it. This will also serve as a good way of getting to know the XML Loader and the data model.</p>
<p><a class="external" href="http://vegbank.org/erd">http://vegbank.org/erd</a> is a good resource for learning the model (click on a table to link to the data dictionary)</p>
Bug #2405 (Resolved): Logout is broken
https://projects.ecoinformatics.org/ecoinfo/issues/2405
2006-04-07T05:31:35Z
Michael Lee
mlee@nceas.ucsb.edu
<p>Oops! You found a VegBank bug.</p>
<pre><code>Sorry, but the last thing you tried to do did not work.</code></pre>
<pre><code>Please try again right now by reloading/refreshing this page or going back.<br /> If it is still broken, please try again later.<br /> This problem has been reported to the developers.</code></pre>
<pre><code>Thank you for your patience.<br /> <a class="email" href="mailto:help@vegbank.org">help@vegbank.org</a></code></pre>
<pre><code>ERROR MESSAGES:</code></pre>
Bug #2395 (Resolved): Fix Backup Strategy -- delete extra backups from time to time
https://projects.ecoinformatics.org/ecoinfo/issues/2395
2006-04-05T17:37:50Z
Michael Lee
mlee@nceas.ucsb.edu
<p>Our current backup strategy is to backup the database every night (which is good). However, we don't delete any old backups, so we will inevitably run out of space. This needs to be fixed. We certainly don't need backups for EACH day the database has existed, but having some older copies might not be horrible either. I propose:</p>
<p>Keep all the backups for the last month.<br />Keep one weekly backup (or every 10 days, whatever is easier) from one month ago to 6 months ago.<br />Keep one monthly backup for anything older than 6 months.</p>
<p>In addition, we need to make sure that this backup is not just to VegBank- a copy needs to be distributed, either by email or network to an alternate machine. I don't know the current status of that "other-machine/off-site" backup, but it would be nice to know. PMark?</p>