User Tools

Site Tools


software:metrics

This is an old revision of the document!


Metrics

Sources

There are a few places that metrics can be found for membership and classes:

Neon CRM

  • source of truth for memberships, membership types, membership-related transactions, and clearances earned for individual members
  • also source of truth for which classes were offered to the public, plus registrations for those classes
  • Neon CRM has several reporting features, but they suffer from “black box” syndrome and are very difficult to dig into to understand and trust the results.

Airtable

  • Stores some metadata about classes that may not be present in Neon (e.g. which instructor taught the class, intended frequency of class to be run)
  • Can be used to create public “views” of data, e.g. the tool status tables at equipment. Depending on the context and counter to the name, views can also be read-write and affect the data.
  • We have a single (admin) account for Airtable - be advised that changing innocuous things like column names will break various automation systems in mysterious ways.

Google Sheets

  • The Master Instructor Hours and Clearance Log feeds into payroll for instructors and includes info about who passed/didn't pass for classes that provide clearance on the tool.
  • The Protohaven Welcome and Waiver Form (Responses) sheet is a timestamped and somewhat coherent log of members signing in at the front desk, including info about guest sign-in, waiver acceptance, clearances at the time of sign-in etc.
  • Both of these sheets are fed by google form submissions; my long term expectation is that these sheets are converted to instead use Airtable so our data is more centralized and easy to operate on.
  • There's also Protohaven Class Feedback (Responses) of customer feedback on classes - this is what's anonymized and sent to Discord's class-automation channel.

Analysis

There's some prior work on understanding the ebb and flow of membership:

Various scripts in the Scripts & Colabs shared drive folder are hacked-together attempts to pull data from Neon, Airtable, and Sheets.

The Class Registration Data Extraction colab pulls event and registration data from Neon and dumps it to CSV for later analysis. This can be useful for detecting occurrence/frequency of particular classes, occupancy and earnings etc. It doesn't include any walk-ins (e.g. techs backfilling not-quite-full classes) which are charged through Square. The Identify paying, active members for a given period colab crawls through all accounts on Neon CRM, accumulates membership information, and uses the data to compute a histogram of new, leaving, returning, and sustained membership on a month-to-month basis. It's not super optimized and takes several minutes to run. Please be careful when executing scripts here, as some of them actually mutate data when you run them.

Materialization of these scripts from previous runs:

  • Membership analysis sheet using the output of the paying/active members colab above, plus a couple of rules and charts to track how we're doing on acquisition

Creating New Metrics

Some questions to answer before creating a new metric or dashboard:

  • What questions do we always need to answer, on a regular enough basis that it's worth paying (time, attention, effort) to maintain a dashboard/pipeline for? What is this frequency/basis? Is the audience just the M&P committee, the whole board, or broader?
  • For each of these questions, can it wait a few minutes, or do we need an answer at any moment? (operational metrics vs higher-level reporting)
  • What would be an ideal “data view” or schema that would support deeper, ad-hoc analysis to answer various questions that emerge from the questions in #1?
software/metrics.1717606009.txt.gz · Last modified: 2024/06/05 16:46 by smartin015