Common use of De-duplicator Clause in Contracts

De-duplicator. The De-duplicator module described in 2.1.9 is also available as a standalone web service accessible from ▇▇▇▇://▇▇▇.▇▇▇▇.▇▇/soaplab2-axis/#ilsp.ilsp_deduplicatormd5_row. The service has two mandatory parameters: 1. The input denotes a file containing a list with URLs to the files to be de-duplicated. 2. The inputType denotes the type of the files to be de-duplicated. These files could be text or TO1 XML files similar to the ones generated by the FMC. The service also has two optional parameters: 1. minimumTokenLength During the calculation of the page profile, all tokens equal or shorter than this value are discarded. The default value is 2. 2. quantValue. Tokens with frequency (after quantization) below this value are discarded. The default value is 3. The output is a text file containing a list with URLs pointing to the files that have remained after de-duplication.