Apache & NGINX log files consolidated into a fully Normalized Database

Proper Python & NGINX formats released with Factory Method implementation

Python handles File Processing & MySQL or MariaDB handles Data Processing

log-files-to-mysql consists of two Python Modules & one Database Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.

Quickly convert a folder of HTTP log files into a normalized database. Process unlimited files, from multiple domains, from multiple servers with complete Data Lineage. Small codebase & simple setup.

Runs on Windows, Linux and MacOS. Database runs on MySQL and MariaDB.

multiple LogFormats & ErrorLogFormats can be loaded and multiple Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

IP Geolocation data integration using MaxMind GeoIP2 Python API provides IP country, subdivision, city, system organization, network and coordinates information stored and normalized into 6 Database Schema tables.

User-Agent data integration using User-Agents provides browser, device and operating system information stored and normalized into 11 Database Schema tables.

Database Schema designed to accommodate unlimited Servers & Domains. Step-by-step guide for easy installation.

Screenshots of Process Messages and Benchmarks for MariaDB version 11.8 and MySQL version 9.2.

HTTP Log Management and Visualization

Express handles Data Querying & ECharts handles Data Visualization of MySQL or MariaDB Schema

mysql-to-apache-echarts consists of Express web application frameworks with Drill Down Capability & Apache ECharts frameworks for Log Data Visualization in charts, reports & data analysis interfaces.

mysql-to-apache-echarts is designed for MySQL or MariaDB databases created by mysql-schema-http-logs and populated by log-files-to-mysql that started as a Python script to LOAD DATA as the database design took shape.

mysql-to-apache-echarts is a Web interface that consists of Express.js web application frameworks with Drill Down Capability & Apache ECharts frameworks for Log Data Visualization in charts, reports & data analysis interfaces of MySQL or MariaDB databases created by mysql-schema-http-logs and mysql-schema-system-logs and populated by log-files-to-mysql.

Application is in development with initial release scheduled for May.