CSE115 Project 1 | Python | WEB | DATA Visualization
CSE115 Project 1
Part 4 (front-end and web server) requirements
Submit a ZIP file of your code (like with lab activities and lab quizzes) by Friday May 1st at 6:00 PM, worth 48 out of the project’s 60 total points.
NOTE: You have a limited number (5) of AutoLab submissions. Use them wisely. Your LAST submission counts.
Languages: JavaScript and Python
Topics: lists, dictionaries, loops, conditionals, reading CSV files, writing CSV files, HTML, JavaScript front end, Bottle web server, JSON, AJAX (lectures up to and including Monday, April 6)
Overview
In this last part of the project you will put the pieces from earlier parts of the project to create a small web application. There are several pieces you need to complete; each is relatively small, but you need to work incrementally to ensure that all the pieces work together in the finished product.
If you did not quite complete the functions for parts 1, 2, and/or 3 keep working on them. Remember that the bulk of the points come from your final project submission.
Your finished web application must create three graphs. Because the OpenBuffalo towing dataset is updated daily, the data your program downloads (and is then shown in your graph) can change. So while your graphs should look similar to those below, they will not be identical. You can use AutoLab to check your code for the first 3 parts and then use those results to help verify your graphs are showing the proper data. The first graph is a scatter plot showing the number of tows on each day of the month:
The second graph is a pie chart showing how the police districts were responsible for tows: The third graph is a line graph showing why cars were towed and how those reasons change from month to month:
Web server (Python)
Define your Bottle web server so it has five routes:
- The “/” must serve up the HTML file as a static file.
You may use any name you want for the remaining routes. Best practices encourage selecting route names that are meaningful and descriptive within the context of the application.
- A route to serve up the JavaScript code file as a static file
- A route to serve up the scatter plot data as a JSON blob
- A route to serve up the pie chart data as a JSON blob
- A route to serve up the line graph data a JSON blob
HTML file
You will need to write an HTML file for your application. Inside your HTML file’s <head> element, it will need 2 <script> elements to load in the needed JavaScript files. The first <script> element should cause the Plot.ly library to be loaded. The URL for this is:
The second <script> element will be the JavaScript code you wrote for this application. Since this file is on the same web server as your HTML file, the second <script> element’s src property should use the route specified in your web server (do not include a protocol or server).
Your HTML file also needs a <body> element. The <body> element will need to include an onload property. This should call a function in your JavaScript code which sends 3 data requests to the web server: one for each of your graphs. When the server responds to each request, you will need code that creates the graph using that data. As an example, the <body> element in my code is:
<body onload=”getData();”>
Inside the <body> element, you will need to include 3 <div> elements. Each <div> element will need to include an id property and be assigned a unique value. You will need to use these ids to display your graphs.
Data processing (Python)
You will also need additional code to process the data.The functions required for parts 1, 2, and 3 of the project will make this process much simpler. You should make certain they are available for this part of the project (copying them in as needed).
Graph #1 – the scatter plot
This chart must show the number of tows that occurred on each day each month. After reading the dataset from your CSV file, count the number of dictionaries recorded for each of the 31 possible days.
Use a layout to give the chart a title and labels for the x- and y-axes. For more information, use this link to Plot.ly’s scatter plot documentation.
Graph #2 – the pie chart
This chart breaks out the tows by the police district in which it occurred. Even though the graph shows percentages, it uses the number of tows in each district. For a pie charts Plot.ly needs two lists: one with names (“labels”) and one with the data (“values”). The labels will be the names of the police districts: District A, District B, District C, District D, and District E. To calculate the values, you will need to read in the complete data set from your CSV file. You will then want to work with each district in the order they appear in your labels list. For each district, get the list of matches using ‘police_district’ as the key and the district’s name as the value. Add the length of that list to your list of values.
Use a layout to give the chart a title. For more information about pie charts, use this link to the Plot.ly documentation.
Graph #3 – the line graph
This must show the number of tows broken down by the description of why the tow occurred. Within each tow description, you will need to plot how often that was the reason for a tow in each month. The way to do this is to read the complete data set from the CSV file and get the list of all descriptions. You can then get a list of dictionaries which matches that description (use get_matches) and use that to get a month-by-month count (the value returned by your count_by_month function). Remember to add a name to each set of data displayed in the graph.
Use a layout to give the chart a title and labels for the x- and y-axes. For more information, use this link to Plot.ly’s line graph documentation.
Fetching and caching OpenBuffalo data
The following function uses your get_data, minimize_dictionary, and write_cache functions to get data from the OpenBuffalo site. In doing this, it includes two additional optimizations. The function begins by checking if the cached_data.csv file exists. When the file exists, it will not request data from OpenBuffalo or update the cache. This minimizes the likelihood of your application going over any resource limits. Second, it uses a query string that the OpenBuffalo API provides to limit the request to 10000 records. Note that this query string is specific to this API and may not be usable everywhere.
Please see https://piazza.com/class/k5e51319xsy76d?cid=879 for a change this will require you make to your minimize_dictionary function.
You should include this in your code and call this function just before starting the webserver.
import os.path
import cache
# This assumes that your functions from parts
2 & 3 are in a file named cache.py
def load_data( ):
csv_file = ‘cached_data.csv’
if not os.path.isfile(csv_file):
query_str = “?$limit=10000”
url = “https://data.buffalony.gov/resource/5c88-nfii.json” + query_str
data = cache.get_data(url)
data = cache.minimize_dictionaries(data, [‘tow_date’, ‘tow_description’, ‘police_district’])
cache.write_cache(data, csv_file)
JSON and AJAX
JavaScript front-end and Python back-end run on different machines. (JavaScript is executed by the machine of the person looking at the page; the Python code is executed by repl.it’s servers.) This means that we cannot communicate directly from JavaScript to Python (or vice versa). We instead rely on an approach called AJAX to allow the two sides to coordinate their work.
This process begins when the JavaScript front-end needs data from the Python back-end. To make this request, it must do two things:
- Send a request to the repl.it server requesting the data;
- Tell the JavaScript system what function should be called if/when it receives this response.
You will need the ajaxGetRequest function to accomplish both of these tasks. We will discuss how to use this function during lectures, you should copy-and-paste this function into your JavaScript code:
// path — string specifying URL to which data request is sent
// callback — function called by JavaScript when response is received
function ajaxGetRequest(path, callback) {
let request = new XMLHttpRequest();
request.onreadystatechange = function() {
if (this.readyState===4 && this.status ===200) {
callback(this.response);
}
}
request.open(“GET”, path);
request.send();
}
You will need to provide two arguments for each of your 3 (or more) calls to ajaxGetRequest. An example of this call is:
ajaxGetRequest(“/towsByDay”, showScatter);
The first argument (“/towsByDay”) is the route on the repl.it server where the request is sent. For this to work, the Python back-end will need a function with a matching annotation
(e.g., @bottle.route(“/towsByDay”)). The second argument (showScatter) must be the name of the JavaScript function you wrote. JavaScript automatically executes the second argument (showScatter), but only when it receives a response from repl.it. That function should include one parameter. This parameter will be the data the back-end sent over the Internet to respond to our request. As an example, the first bit of my function looks like this:
function showScatter(response) {
// Data is always sent in JSON, so we must first convert it
let data = JSON.parse(response);
// Now write the code that uses data to create the scatter plot
Finally, where do we make the calls to the ajaxGetRequest function? For this project, these should be in the JavaScript function called in the <body> tag’s onload property. In the example above the function getData would have all of the calls to ajaxGetRequest.
GRADING
For your final project submission, autogrades in AutoLab will score the functions specified in parts 1, 2, and 3. The functionality for part 4 of the final project will be manual graded by the TAs in the weeks after the project deadline. As before, you are limited to 5 submissions on AutoLab and it is your last submission that counts: that is the autograder score we will use and the submission that will be manually graded by the TAs.
SUBMISSION INSTRUCTIONS
In order for us to grade your project, you must submit all of your .html, .js, and .py files. We will download and execute your project for the manual grading, so make certain everything runs before you submit.
Code organization
It is important you follow these instructions
so that the autograders can run and you earn those points.
Your Python code MUST
be separated into 3 files and the files MUST have the following names:
cache.py – this file must contain ALL OF THE FUNCTIONS FROM PARTS 2 and 3. Do not include any web server code and the file cannot import or use the bottle library.
backend.py – this file must contain ALL OF THE FUNCTIONS FROM PART 1. You can include the functions from part 4 that process the data, but this is optional. Do not include any web server code and the file cannot import or use the bottle library.
main.py – this file should include all of the Python code that defines the web server’s behavior. This file should have the following basic structure:
import bottle
import json
import cache
import backend
@bottle.route(…)
# …all the routes and handlers defined here…
import os.path
def load_data( ):
# You should use my code for loadData()
load_data()
bottle.run(…)
Note the import cache and import backend lines. You will need to include the name of the module when you call functions that are in those files. For example, when loadData wants to call the get_data function, it write the call as cache.get_data(url).
Submission
To submit, export the project files from repl.it as a zip file. If you can, please remove the following files from the ZIP file you submit:
cached_data.csv
poetry.lock
pyproject.toml
requirements.txt
but only do this if you are confident you know how. The most important thing is that we can download and run your submission.
Feedback
Autolab will give two kinds of feedback:
- An ungraded checklist showing how well your code meets (some of) the explicitly stated expectations spelled out above in this document. The first bit of the feedback will look something like this:
******************************************************************************
*** File names and quantities ***
******************************************************************************
PASS: I expected to find 1 HTML file. I found 1:
index.html
PASS: I expected to find 1 .js file. I found 1:
permits.js
PASS: I expected to find 1 file named ‘main.py’. I found 1:
main.py
PASS: I expected to find 1 file named ‘cache.py’. I found 1:
cache.py
PASS: I expected to find 1 file named
‘backend.py’. I found 1:
backend.py
Please verify that all of the checks are labeled “PASS”. This is needed for the autograders and makes the manual grading go much faster. We gave you lots of leeway for how to structure your code for part 4, but may not be able to find and grade code if you move things around.
- A report on the automated grading of the functionality of parts 1, 2, and 3. Each part is graded out of 8 points.
Overall Project Grading
| Project Part | Part 1 submit | Part 2 submit | Part 3 submit | Final submission | Overall points |
| Part 1 | 4 | 10 | 14 | ||
| Part 2 | 4 | 10 | 14 | ||
| Part 3 | 4 | 10 | 14 | ||
| Part 4 | 18 (manually graded) | 18 |
