Preprocessing. The log data are formatted as Apache log files.1 We fil- tered the raw data as follows: We removed all requests that did not result in a successful response (status codes starting SIGIR ’16, July 17-21, 2016, Pisa, Italy ◯c 2016 ACM. ISBN 978-1-4503-4069-4/16/07. . . $15.00 DOI: xxxx://xx.xxx.xxx/10.1145/2911451.2914667 1See xxxxx://xxxxx.xxxxxx.xxx/docs/1.3/logs.html for a def- inition of the format. Figure 1: Screenshot of small portion of the filtered data with masked IP addresses in the first column. with 3 or higher); all requests that are no GET requests; and all requests for images and other files that do not result from a navigational process.2 In addition, we removed all re- quests that supposedly come from web bots, using the regu- lar expression .*(Yahoo! Slurp|bingbot|Googlebot).* on the log entry. We anonymized the data by taking the fol- lowing measures: • We replaced all occurrences of the same IP address by a unique random identifier (a 10-digit string). • We removed the last part of each log entry – the User- Agent HTTP request header – which is the identifying information that the client browser reports about it- self.3 • If the referrer is a search engine, we removed every- thing after the substring /search?. We are aware that queries can provide valuable information about pages in the domain [2], but queries are also known to po- tentially be personally identifiable information [1]; for that reason, we will postpone a decision on releasing fil- tered query information, and first gain experience with the external usage of the data without search queries. • We removed requests for URLs that only occur once in the 3-month-dataset to reduce the chance of unmask- ing specific users. This is an additional security step since extremely low-frequent URLs are highly specific and therefore often unique for a person. The effect of each of the filtering steps is shown in Table 1. The information that is retained per entry is: unique user id, timestamp, GET request (URL), status code, the size of the object returned to the client, and the referrer URL. A xxx- ple of the resulting data is shown in Figure 1. The sample illustrates that the content (URLs and referrers) is multi- lingual: predominantly Dutch, and English and German in smaller proportions.
Preprocessing. We start from a set of measured variables X at measurement locations K to learn a local causal graph that is valid for all locations. Trivially, we could run the FCI algorithm. This, however, is suboptimal due to several reasons. First, the measurements are not IID, violating one of the basic assumptions of the FCI algorithm. Second, by neglecting the spatial structure, we would be neglecting a lot of information that could potentially be useful (Xxxxxxx et al. (2000)). To avoid this loss of information, we construct upstream variables U as outlined in def. 1, using the mean as function f . We stress that this choice depends on the application. For example, for a system with currents of largely differing discharge volume, the weighted average might be a better choice. Further, we note that not all locations have a preceding location which results in locations with an incomplete set of variables. In this work, we wanted to evaluate our general approach, which is why we decided to exclude missing data imputation as a potential influence on modeling performance. In principle, however, the missing data problem could be tackled by any appropriate strategy, including regression imputation and Bayesian estimation (Xxxxxx (2010)). The spatial structure of the system could offer additional information here that could allow for better imputation of missing data. Note that our strategy of excluding locations with an incomplete set of variables slightly reduces the size of the data set. Our preprocessing strategy is outlined in alg. 1. Algorithm 1: Current data preprocessing ∈ / ∅ Input : Set of measured variables X at locations K in a system with directional currents, with KS being the set of locations x X for which Pre(k) = { } Output: Set of variables Xr = U , O, R, I at locations Kr 1 Partition variables into subsets I, O and R following def. 1; 2 repeat ∈ 3 Select an unvisited measurement location k KS and calculate U (k) following def. 1, using the average as the function f ; 4 until All measurement points k ∈ KS have been visited; 5 Remove entries of locations k ∈ K for which Pre(k) = ∅;
Preprocessing. A number of preprocessing steps are required. These steps standardize the geographic references and create a number of look-up tables that greatly speed the complex data processing. The following pre-processing tasks are performed: • Import and standardize AVL files, • Create stop location table • Update off-board stop location table • Create (or update) the quarter-mile look-up table • Create subsidy table • Link the subsidy table to ORCA cards (CSNs) • Hash the CSNs and Business IDs in the subsidy table, maintaining the link between the subsidy table and the hashed CSNs • Preprocess date and time values in the transaction data • Remove duplicate boarding records. Each of these tasks is described below.
Preprocessing. In order to fuse the input data sets together, geolocate the transactions, and then create the origin/destination and transfer files, a number of preprocessing steps must first be performed. These steps standardize the geographic references and create a number of look-up tables that greatly speed the complex data processing. The following are the pre-processing tasks: • Import and standardize the AVL files • Create a stop location table • Create a table that correlates the ORCA transaction record’s directional variable (i.e., inbound or outbound) with the cardinal directions used by the transit agency’s directional variable (i.e., north/south/east/west) • Create (or update) the quarter-mile look-up table • Update the off-board stop location table • Preprocess ORCA transactions data and reformat date and time variables • Create a subsidy table • Link the subsidy table to ORCA cards (CSNs) • Hash the CSNs and Business IDs in the subsidy table, maintaining the link between the subsidy table and the hashed CSNs • Remove duplicate boarding records. These tasks are described below. Schema for each of the data sets are presented in Appendix C.
Preprocessing. × Processing of CT scans can be a memory-intensive task, particularly for high-resolution data. For this reason, some of the segmentation tasks described in this work use sub- sampled data. To subsample the data, the image size was reduced by block averaging to 256 256 voxels in the X-Y plane with the number of slices reduced such that the data were isotropically sampled. Linear interpolation was used to determine gray-values between voxel locations. This strat- egy does not attempt to apply a consistent image spacing for all images but rather aims to retain the best resolution possi- ble for each individual image. In this way, we hope to achieve the highest chance of success for each image in sub- sequent processing tasks. In order to reduce memory consumption in processes where full resolution data are preferred, the scan size was reduced by excluding image regions outside the lungs (after lung segmentation has taken place). A bounding box around the segmented lungs was constructed, with a margin of 5 voxels on each side. Data outside this bounding box were discarded. The resulting smaller image is referred to in this work as the bounded image.
Preprocessing. Alphabet Size We first show that in Definition 3.1 it is sufficient to consider random variables U and V over X , i.e., the alphabet of U and V need not be larger |
Preprocessing. Before the Bayesian analysis, we cleaned the data and visualized general tendencies present in the data as summary plots using the tidyverse package system in R (Xxxxxxx et al., 2019). In the data-cleaning process, we had several criteria for exclusion. The first criteria was participants’ native language: we excluded participants whose native language is not Turkish. The second criteria was their accuracy in practice items: if they give wrong answers to more than half of the questions, we excluded them from the analysis. We also excluded participants that answered the questions too fast, that is below 200 milliseconds. Finally, we excluded participants with too many inaccurate answers in control conditions. We did not include missing data points or exclusions in our analysis and assumed that data were missing completely at random (Xxx Xxxxxx, 2018). In this thesis, we do not report the rates of missing data, but our raw data is available.
Preprocessing. For the visualization, the EEG was bandpass filtered between 0.3 and 40 Hz. For deep learning classifiers and cluster encoders, the EEG was bandpass filtered between 1 and 40 Hz, re-refer- enced to the common average, and normalized by dividing by the 99th percentile of the absolute amplitude. All filters were implemented in Python as 5th order Butterworth filters using scipy.signal (Xxxxxxxx et al., 2020) and zero-phase filtering.
Preprocessing
