Metacat

    <div class="toc1"><a href="#LoadTesting">Load Testing</a></div>

      <div class="toc2"><a href="#LoadTestCodeFiles">Load Test Code Files</a></div>

      <div class="toc2"><a href="#GeneratingDictionaryFiles">Generating Dictionary Files</a></div>

      <div class="toc2"><a href="#InsertLoadTest">Insert Load Test</a></div>

      <div class="toc2"><a href="#SQueryLoadTest">SQuery Load Test</a></div>

      <div class="toc2"><a href="#ReadLoadTest">Read Load Test</a></div>

      <div class="toc2"><a href="#TestDriverScript">Test Driver Script</a></div>

    <div class="toc1"><a href="#ProfilingMetacat">Profiling Metacat</a></div>

  <a name="LoadTesting"></a><div class="header1">Load Testing</div>

  <p>This section dicussed the available load testing code and its usage.</p>

  <a name="LoadTestCodeFiles"></a><div class="header2">Load Test Code Files</div>  

  <p>The code to do load testing is located in the metacat project in this directory:</p>

     <div class="code">&lt;metacat_src&gt;/test/stress-test</div>

  <p>The test code files are written in python for the following tests:</p>

  <ul>

  <li> read - read-load-test.py </li>

  <li> insert - insert-load-test.py </li>

  <li> squery - squery-load-test.py </li>

  </ul>

  <p>While these can be run directly from the command line, there is also a driver 

    file written in bash for convenience named: load-test-driver.sh</p>

  <p>The insert and squery tests rely on the following template files respectively: </p>

    <div class="code">insert.xml.tmpl and squery.xml.tmpl</div>

  <p>The insert and squery tests rely on dictionary files to create unique document IDs. 

    These files are generated using a shell script named:</p> 

    <div class="code">generate-dictionaries.sh</div>

  <p>Each of these files will be discussed in the next sections</p>

  <a name="GeneratingDictionaryFiles"></a><div class="header2">Generating Dictionary Files</div>  

  <p>The insert and squery tests (see following sections) will need to create unique document IDs to avoid conflicts and to bypass caching mechanisms. The dictionary files are created by running:

     <div class="code">./generate-dictionaries.sh</div>

  <p>This will create a separate file for each letter of the alphabet that looks like:</p>

     <div class="code">dictionary-a.txt, dictionary-b.txt, etc.</div>

  <p>Each file will contain all the five letter word combinations that start with the 

    letter associated with that file. You should run this script right away, as it takes a little 

    time to run.</p>

  <a name="InsertLoadTest"></a><div class="header2">Insert Load Test</div>      

  <p>The insert load test is run via a python script with the following usage:</p>

     <div class="code">./insert-load-test.py &lt;letter&gt; &lt;iterations&gt; &lt;interval&gt; &lt;host&gt; 2>&1 &</div>

  <p>Where:</p>

  <ul>

  <li> letter - the letter of the dictionary you want to use to generate doc IDs.</li>

  <li> iterations - the number of inserts you would like the test to perform.</li>

  <li> interval - the delay in seconds between each insert. You can enter a decimal for 

    less than one second.</li>

  <li> host - the server that is running the instance of metacat you are load testing. 

    You should not be running the test drivers on the same machine as metacat, since that 

    could affect the outcome of the load test.</li>

  </ul>

  <p>The insert test will iterate through the dictionary for the letter you have specified. 

    For each word, it will create a doc ID that looks like:</p>

     <div class="code">&lt;word>&lt;epoch_date&gt;.&lt;epoch_date&gt;.1</div>

  <p>For instance, if the test started at epoch date 123914076495 and the 67th word in the 

    dictionary file (for letter c) is caacp, your doc ID will look like:</p>

     <div class="code">caacp123914076495.67.1</div>

  <p>This docid is subtituted for each values of @!docid!@ in the insert template at:</p>

     <div class="code">insert.xml.tmpl</div>

  <p>Each doc will then be inserted into Metacat using the metacat.py interface file.</p>

  <p>Output will be written to a file named:</p>

     <div class="code">insert-&lt;letter&gt;.out</div>

  <p>Note that you can run several of the insert tests at the same time. You should run 

    each against a different letter to avoid doc ID naming conflicts and to be able to 

    view the output from each test in a different output file. See the load-test-driver.sh 

    for some examples.</p>

  <a name="SQueryLoadTest"></a><div class="header2">SQuery Load Test</div>  

  <p>The squery load test is run via a python script with the following usage:</p>

     <div class="code">./squery-load-test.py &lt;letter&gt; &lt;iterations&gt; &lt;interval&gt; &lt;host&gt; 2>&1 &</div>

  <p>Where:</p>

  <ul>

  <li> letter - the letter of the dictionary you want to use to generate doc IDs.</li>

  <li> iterations - the number of squeries you would like the test to perform.</li>

  <li> interval - the delay in seconds between each squery. You can enter a decimal for 

    less than one second.</li>

  <li> host - the server that is running the instance of metacat you are load testing. 

    You should not be running the test drivers on the same machine as metacat, since 

      that could affect the outcome of the load test.</li>

  </ul>

  <p>The squery test will iterate through the dictionary for the letter you have specified. 

    For each word, it will create a query by substituting the dictionary word for every 

    instance of @!search-word!@ in the squery template at:</p>

     <div class="code">squery.xml.tmpl</div>

  <p>Each of these queries will be run against Metacat using the metacat.py interface file. 

  By changing the query for each word, we insure that we do not get cached query results 

  back from Metacat, which would not cause a significant load.</p>

  <p>Output will be written to a file named:</p>

     <div class="code">squery-&lt;letter&gt;.out</div>

  <p>Note that you can run several of the squery tests at the same time. You should run 

    each against a different letter to avoid doc ID naming conflicts and to be able to 

    view the output from each test in a different output file. See the load-test-driver.sh 

    for some examples. If you are going to run a test against the same letter more than 

    once, you will need to restart the instance of Metacat being tested to clear the query 

    cache on that system.</p>

  <a name="ReadLoadTest"></a><div class="header2">Read Load Test</div>     

  <p>The read load test is run via a python script with the following usage:</p>

     <div class="code">./read-load-test.py &lt;letter&gt; &lt;iterations&gt; &lt;interval&gt; &lt;host&gt; 2>&1 &</div>

  <p>Where:</p>

  <ul>

  <li> letter - the read test does not use a dictionary. The letter helps us insure that 

    each test reads a different document, and writes to its own output file.</li>

  <li> iterations - the number of reads you would like the test to perform.</li>

  <li> interval - the delay in seconds between each read. You can enter a decimal for 

    less than one second.</li>

  <li> host - the server that is running the instance of metacat you are load testing. 

    You should not be running the test drivers on the same machine as metacat, since 

    that could affect the outcome of the load test.</li>

  </ul>

  <p>The read test will create a doc ID that looks like:</p>

     <div class="code">readtest-&lt;letter&gt;&lt;epoch_date&gt;.1.1</div>

  <p>It will do a single insert using that doc ID and the template at:</p>

     <div class="code">insert.xml.tmpl</div>

  <p>It will then do a read of that document from Metacat using the metacat.py interface 

    file for the number of iterations you have specified.</p>

  <p>Output will be written to a file named:</p>

     <div class="code">read-&lt;letter&gt;.out</div>

  <p>Note that you can run several of the read tests at the same time. You should run 

  each against a different letter to avoid doc ID naming conflicts and to be able to 

  view the output from each test in a different output file. See the load-test-driver.sh 

  for some examples.</p>

  <a name="TestDriverScript"></a><div class="header2">Test Driver Script</div> 

  <p>There is a very simple driver script that allows you to easily run multiple instances 

  and combinations of the different load tests called:</p>

     <div class="code">./load-test-driver.sh</div>

  <p>Uncomment the tests you want to run. </p>