Project Deep Dive — Network Health Survey Reports
Overview
The objective of this project was create an online dashboard for school administrators and researchers to view reports based on survey data from their schools. These data were aggregated, analyzed, and exported to be fed into various visualizations, and these visualizations would be organized based on “constructs” and “measures”, i.e. groups of questions. Each construct page also included related data such as Ns (number of responses to the questions), the list of possible responses to the questions, as well as some information on how to interpret the visualizations.
Report generation was semi-automated, leaving the data analysis and computation largely a manual process. A workflow was initialized which aggregated survey data by school and sent it to our analytics team. Once data had been analyzed and transformed into a programatically-friendly schema, it was ingested into our data warehouse. From there, the workflow because automated once more, transforming the data from our data warehouse into JSON where it could be read by the report app. Then one final look-over by our analysts before the report was marked as “ready to view”.
While the construction of the Network Health Survey itself was a core part of this work and an obvious prerequisite, work on the survey happened separately, before work on the report app had started. This project had a tight five-month deadline. Four sprints were dedicated to research, planning, and scaffolding, with the rest of the time dedicated to development, followed by two sprints of testing, bug fixes, and final enhancements.
Context
Data dashboards and reports are nothing new at the Carnegie Foundation. My team has created similar things in the past for various other projects, and I gladly drew on this past experience when planning this new app.
For our Carnegie Math Pathways project, a new developmental math curriculum for colleges and universities, we built the Productive Persistence surveys, which administered three times over the course of the semester/quarter. These data were aggregated, analyzed, and presented in a report per course, later rolled up to the school level to be presented in an “institutional report” viewable by school administrators and district leads.
For our Student Agency Improvement Communities project, networks of schools (usually K-12) organized around implementing improvement science, we built the Student Agency survey, where questions were broken into the various subjects taught at the school, and measures such as Value, Growth Mindset, and Belonging were evaluated. Surveys were administered one or more times per academic year, to be determined by that district’s lead. A similar aggregation process was employed here for the reports, presenting data at the school level for the district lead.
These were two, nearly-year-long projects, that resulted in lots of learning for me and my team (and hopefully for the schools involved as well). Lots of technical infrastructure was also put into place to enable these large projects, and I would take advantage of this when developing this new report app.
Part 1: Conceptualization
Requirements-gathering
Lots of meetings were required to gather the business requirements for this project. For one, the stakeholders were slightly different than before. In the past, the report viewers were teachers, school administrators, and districts leads, who needed clear, actionable data, as well as resources in the form of interventions they could take to act on the data and start making improvements. Our audience for these reports would be school administrators in addition to education researchers. Some of these districts even had their own analytics department. So this meant analytical knowledge and capacity was a bit higher for this audience. Language used throughout the app would reflect this, and resources, instead of being interventions, would focus on how to interpret the graphs and explaining our analytical process.
The report app would be divided into several high-level “constructs”, categories of questions, and within these constructs would be a second level of “measures”, smaller groups of questions. Certain graphs would be displayed at the construct level, and upon clicking a data point, you would be taken to the appropriate measure, where you would see another set of graphs. The object of this was to create a “drill-down” user experience, where users would begin at the highest level of data and then investigate their school’s results, drilling down into the measures to get more specifics.
Technical Decisions
Both of these concerns, the more analytical audience and the desired layout and navigation, lead me to determine that this app would need a high level of interactability. Navigation between constructs and between measures would have to be quick and seamless. Additionally, vanilla versions of our graphs wouldn’t cut it. Historical data was to be included, graphs would need to be coded differently for certain constructs/measures, and the graphical display itself would need to be altered to increase readability.
Frontend
Based on these requirements, I chose to create a single page app using React for the frontend, along with d3 as our data visualization library. And this was to be embedded within our existing “survey portal” application, which orchestrated creation, scheduling, and delivery of our surveys.
React is a library that my team and I were already familiar with, so it was the most sensible choice for the frontend. We preferred the one-way data binding model as opposed to Angular’s two-way data binding, and state management, if it was needed, was easy enough with Redux. We also liked its simplicity and modularity; we could easily integrate it into an existing app and bring in any additional libraries we needed. We were not familiar with Vue or Ember so those libraries didn’t factor into my decision.
d3 was another clear choice, given that modifications to the traditional visualization structures would be necessary. My team has also used it in our previous report development. The big issue here would be that both React and d3 want to manipulate the DOM, so some creativity was required to overcome this.
Developing a single page app meant we could do all the “heavy lifting” on the client side and avoid lots of trips to the server. We would also do a single big query to the server for the report data. This meant, after the initial page load, navigation was ultra fast. Since the Survey Portal was a legacy app with a Bootstrap/jQuery frontend, I also needed to figure out how to embed this report app in a way to make development, deployment, and viewing seamless between the two.
Backend
The last piece of this puzzle was the backend. How would we gather the survey data, which was hosted in a separate survey application (which we call the Survey Tool) and database? How would we reformat the data and hand it off to our analytics team? How would they give the data back to us after analysis? And how would we grab this data for use in the reports? Some sort of data processing pipeline was needed, and one that allowed for a semi-automatic process.
Since the app would be embedded into the Survey Portal, it was obvious that the code required to orchestrate all this would live there. With many moving parts and dependencies, this was clearly a job for our WorkflowPlan library. This is something we developed years ago: an automated workflow processor. Complex jobs, or workflows, were broken up into steps, and each step blocked the next one until it was complete. Steps that failed to complete would pause the workflow and require manual intervention to fix, upon which the workflow could be resumed. Steps could also pass data up to the workflow’s state, and this state was accessible to future steps in the workflow. Additionally, steps could be “offline”, where the workflow would pause, allowing something manual elsewhere to happen; then the user would resume the workflow once this happened.
I could easily take the questions I posed above and convert these into steps for a workflow (or further break them down into additional steps). While I would have loved to keep the whole thing automated, there needed to be some manual pieces to this. Talking with our analytics team, I learned that most of their infrastructure was offline. Data was downloaded locally and analysis was performed on the analyst’s computer. And they would need give it back to us in pieces, as soon as the analyst was done with their work. In addition, due to our tight three-month deadline, there wasn’t enough time to build a cloud infrastructure for the analytics team and expose their scripts via API to automate the whole thing. But the requirements and resulting process were clear enough to form into a workflow.
Part 2: Development
It was clear we had our work cut out for us. We needed to build a whole new app as well as a data processing pipeline. Though we’d done similar pipelines before, this was a fairly new implementation for us, and required lots of reformatting and recoding of data to fit the analysts’ needs.
Frontend
Work on the frontend was prefaced by two big problems: how do we embed a separate React app into the Survey Portal’s frontend in a seamless manner, and how do we get React and d3 to play nicely?
This ended up being a topic that warranted a whole separate blog post, but I will summarize it here.
For the first issue, what I ended up doing was utilizing webpack to combine, transpile, and minify the entire React app into a single “bundle” file. This was a somewhat large Javascript file (about 602k in size) that I included in the Survey Portal’s “report” template. Instead of displaying the old template code with its jQuery scripts, the template was emptied and replace by a single <script> tag that imported this bundle file. Voila! An app within an app.
Upon reflection, a clear improvement here would be to split the bundle into vendor code and my code. This means reconfiguring the webpack configuration to generate separate files. So multiple <script> tags, loading the libraries separately from the React components we’d written. The result would be several smaller Javascript files that would be loaded in parallel instead of a single large file. Faster page loads are always a good thing.
Then came the task of getting the report data to the report app. Normally this can be done with an API call to the backend, but instead of API calls, I decided to “dump” the report data into a global variable made available to the React app. While it wasn’t the cleanest way to do it, it saved us trips to the server and frontloaded the data retrieval into the initial request, making the whole app ultra-fast.
The other issue, React and d3, required some creativity. To solve this, I needed React to render the d3 component and then leave it alone. The d3 component was basically a “black box”, where the d3 code only dealt with the <div> that it was writing to. I wrote a common set of code into a D3Component, then used composition for each graph to pass in its own data and visualization code to a D3Component.
Backend
Though our WorkflowPlan library made orchestration of everything possible, I still needed to write the code for each step. Instead of spawning separate workflows for each report, I decided to create a single workflow that handled all of the reports. This was sufficient for the first few steps, though some clever “parallel processing” would be required for later steps. Even so, I at least had all the steps of the workflow figured out.
The first step would be gathering the data from the Survey Tool. This happened by querying the Survey Tool API for data from this year’s survey. Second came reformatting and recoding of certain values to help with analysis down the road. Third came the data handoff, which was just a large CSV export to a folder in Google Cloud Storage. The workflow paused here, while analysts downloaded the data from Cloud Storage and performed their analysis. Insert analytics magic here (I’ve learned this involves R scripts and a tool called HLM). When the analysts were done, they uploaded their data to our data warehouse, Google Bigquery. Here, I resumed the workflow, triggering the fourth step.
The fourth step was special. Previously, I was working on all data, but now, things needed to be broken down by school in order to generate the reports. I took advantage of “parallel processing”, in the form of Google Cloud Tasks, to spawn a task for each report that would query Google Bigquery for data specific to that school, reform the data into JSON, and save it to the report object. So step four basically spawned thirty-five “mini steps”, one for each school, that did this part. Having separate tasks also gave me the additional benefit of being able to manually re-ingest, re-form, and update a report individually. If new data was uploaded for a specific school, all I had to do was spawn a task just for that school. This was invaluable when it came to final testing and verification.
Part 3: Testing
As it is with all things in software and web development, nothing ever runs perfectly the first time. Bugs, inconsistencies, and edge cases needed to be worked on.
Something early on that happened was a memory issue. One piece of the second reformatting and recoding step was creation of something we called the codebook. A codebook delineates each question of the survey and provides metadata of sorts, such as the question’s construct, measure, question text, text of the possible responses, values of the possible responses, what kind of scale it used, etc. Except there was a codebook for each school, because each school used slightly different terminology in the survey, so this needed to be recorded. When generating the codebook for each school, I thought it would be convenient to store all the codebooks in memory so I could retrieve them quickly. This ended up maxing out the memory of our Google App Engine instance. I refactored the code to make separate API requests to get the codebook for each school. The tradeoff was more network traffic and longer workflow execution time, but I think it was the better choice, since we would save money by not having to scale up our instances.
There were an assortment of misunderstandings that came to light once we started testing. Certain assumptions of the analysis process were made by us that ended being incorrect, and vice-versa for the report generation process. One of these was the need for the reformatting and recoding step. This step was initially not part of the workflow, but my team soon learned that the analytics team lacked time and capacity to do this step. This was addressed with a meeting or two, and only added a few days of dev time.
The majority of testing time was taken up by manual review of each report, scanning through each and every graph, looking for odd outliers or incomplete data or browser errors. These types of issues were largely fixed one by one, and tackled report by report, until we could verify green status for all reports. As I mentioned above, this process was greatly expedited by our ability to requeue generation of reports individually.
One enhancement for the future that I’ve suggested is implementing some automated testing on the frontend. I have not implemented frontend testing before, but using something like Jest, we could test certain business rules within the data as well as test for expected changes in the report. I’ve used stuff like RSpec and Selenium before, and Jest can implement similar types of testing. Here’s to using Jest in the future!
Part 4: Success! Vacation! Reflection…
I was beyond tired when we wrapped this up. Not only did development take a ton of work, both from me and my team, testing did too. After green lights and celebration, I took a much needed vacation. Vacation allowed me to reflect on the project, think about the next steps in my career, and write this post! I’m very glad we wrapped up the project and glad it’s been well-received.
When working cross-functionally and between teams, it’s important to be very clear about your expectations and communicate frequently. That seemed to be the biggest area of improvement needed for our next project. Both our team and the analytics team expressed interest in more frequent check-ins and time devoted to really understanding the work that each team is doing.
I will leave you with a few more screenshots of the app!
