About This Project

Disclaimers

• I do work for the Kentucky Transportation Cabinet but this is not an official KYTC project.
• This is a personal project to help me learn web development (Node.js, HTML, CSS, JavaScript, etc).
• I am not a professional software developer.
• This is my first attempt at web development.
• This site was developed as a Code:You web development project but the data is legitimate.
• The next version of the Node.js ETL (etl_data_ai.js) script required heavy use of GitHub Copilot for troubleshooting so it will be waiting until after my Code:You capstone submission.

About Me

• IT Project Manager
• Data Analyst
• Python ETL Developer
• SQL Developer (Google BigQuery, Oracle)
• BI Dashboard Developer (Looker Studio, Tableau, PowerBI)

Project: JavaScript Server-Side Scripting with Node.js

1. Extract: Download the latest CSV data from source GitHub repository.
2. Transform: Parse the CSV and keep only the essential fields (latitude, longitude, comments, reported date, end date, duration).
3. Enrich: For each record, call the KYTC Spatial API to add updated road attributes (district, county, milepoint, road name, route, etc.).
4. Batch Processing: Process records in batches to avoid overloading the API and write results incrementally to a JSON file.
5. Output & Logging: Save the final enriched dataset as data_v4_final_roadclosures.json and log ETL run details (record count, time, etc.) to a CSV log file.

Project: HTML/CSS/JavaScript User Interface

• Responsive Design: All pages are mobile-friendly and use a consistent navigation bar and color scheme.
• Home (index.html): Dashboard landing page with visual links to all analysis and info pages.
• Analysis by Count (analysis_by_count.html): Interactive bar chart (Apache ECharts) showing the number of road closures by year, with tooltips and dynamic charting.
• Analysis by Duration (analysis_by_duration.html): Interactive bar chart (Apache ECharts) showing total closure duration by year, with tooltips and dynamic charting.
• Map View (analysis_by_map.html): Interactive Leaflet map with marker clustering, showing closure locations and popups with details for each closure.
• Table View (analysis_by_table.html): AG Grid table with filtering, sorting, and CSV download for both full and filtered data.
• Power BI Dashboard (powerbi.html): Embedded Power BI dashboard for official reporting and comparison.
• About (about.html): Project background, methodology, and documentation for learning and transparency.

Project: Python Analysis and ETL

This project, developed in 2024, focused on analyzing Kentucky road closure data using Python. This project started as a Code:You Python capstone project. My goal with the Code:You Web Development course was to develop a better front end for this project and data. I decided to use the tranformed data in the Python repo as my orignal data source but I've used a Node.js Javascript ETL process to further enhance the data and then of course using HTML/CSS/Javascript to develop a much better interface. This is where the JavaScript web development project is pulling its data.

• Data Ingestion: Ingests data from multiple CSV files containing annual road closure data, starting with 2021.
• Data Cleaning: Handles various inconsistencies with timestamps, carriage returns, and parsing latitude and longitude.
• Data Transformation: Transforms and merges each of the annual snapshots into a single, consistent, report-ready dataset.
• Reporting: Generates reports in Jupyter Notebooks and PowerBI to present the results.
• Documentation: Comprehensive documentation of the ETL processes and data analysis methodologies.
• Deployment: Recently developed a GitHub Action to automate the ETL process, maintaining the integrity and timeliness of the report-ready dataset.