- Install Python and Pandas:
pip install pandas - nginx.log and log-parsing.py should in the same directory
- Script should be run in initialized Git repo. Do
git initfor not git tracked directories - Run script with
python log-parsing.py [sort] [column].[sort]and[column]are optional. Columns list can be found in the Variables part of script - nginx.csv and, optionally nginx-sorted-by-{column_name}.csv, are created as the output of the script
- Each nginx-sorted-by-{column_name}.csv is added into a separate commit
The script opens nginx.log file and reads each line in While loop. Each line is passed to function log_parser, which splits the line into lists and assignes a specific list element to a variable. Function returns a list of all fields parsed from a single line of nginx.log. Each returned list is added to parsed_logs list that is being processed by Pandas class DataFrame. CSV file is being created using to_csv method. I had two options: use built-in CSV module or Pandas. After investigation Pandas seemed to me as a simplier solution.
In order to implement sorting, the script accepts arguments (sort and the column name). Sorting is done in descending order. At the beginning of the script there's a check if arguments were passed, if the option and the number of arguments is correct.
Sorting is implemeted using Pandas methods read_csv and sort_values. try...catch construct is used to catch KeyError that is thrown if incorrect column name is passed.
In order to add files to Git, I used subprocess module.
Both nginx.csv and nginx-sorted-by-{column_name}.csv are created even if git repo is not initialized. On the other hand nginx.csv is created in both cases: without initialized git repo and sorting option.
- Make executable:
chmod 770 parser.sh - nginx.log and parser.sh should in the same directory
- Script should be run in initialized Git repo. Do
git initfor not git tracked directories - Run script with
./parser.sh [sort] [column_name] - nginx_sh.csv and, optionally nginx_sh_sorted_by_{column_name}.csv, are created as the output of the script
- Each nginx_sh_sorted_by_{column_name}.csv is added into a separate commit
The logic is similar to Python script. I used While loop for reading each line of nginx.log file and parse the information from it. I used cut for choosing a part of the line and awk for choosing specific elements. awk was chosen because it handles leading and trailing whitespaces and it's easier to choose the correct element. Command substitution assignes the output of the commands inside paranthesis to a variable. Then I use printf with placeholders to write values to the file. \"%s\" in printf is used for req_dev value, so that commas inside the string are not considered as delimiter by Excel (or other text editor). Ready .csv file is saved in Git.
I declared an associative array (dict) in order to store column numbers as values for sorting. Options for the script are passed as arguments. Sorting is done in descending order. Column for sorting is chosen based on the column name passed by a user as an argument. Each sorting creates a separate file and saves it to Git.
- docker build -t nginx:outposts .
- docker run -d -p 8080:80 --name nginx_web nginx:outposts
Dockerfile builds nginx image based on Alpine Linux. Custom HTML file is copied to the working directory /usr/share/nginx/html/, which is default document root for the majority of Linux distributions. Site can be accessed at localhost:8080.