User Tools

Site Tools


software:metrics

Metrics

Sources

There are a few places that metrics can be found for membership and classes:

Neon CRM

  • source of truth for memberships, membership types, membership-related transactions, and clearances earned for individual members
  • also source of truth for which classes were offered to the public, plus registrations for those classes
  • Neon CRM has several reporting features, but they suffer from “black box” syndrome and are very difficult to dig into to understand and trust the results.

Airtable

  • Stores some metadata about classes that may not be present in Neon (e.g. which instructor taught the class, intended frequency of class to be run)
  • Can be used to create public “views” of data, e.g. the tool status tables at equipment. Depending on the context and counter to the name, views can also be read-write and affect the data.
  • We have a single (admin) account for Airtable - be advised that changing innocuous things like column names will break various automation systems in mysterious ways.

Google Sheets

  • The Master Instructor Hours and Clearance Log feeds into payroll for instructors and includes info about who passed/didn't pass for classes that provide clearance on the tool.
  • The Protohaven Welcome and Waiver Form (Responses) sheet is a timestamped and somewhat coherent log of members signing in at the front desk, including info about guest sign-in, waiver acceptance, clearances at the time of sign-in etc.
  • Both of these sheets are fed by google form submissions; my long term expectation is that these sheets are converted to instead use Airtable so our data is more centralized and easy to operate on.
  • There's also Protohaven Class Feedback (Responses) of customer feedback on classes - this is what's anonymized and sent to Discord's class-automation channel.

Analysis

There's some prior work on understanding the ebb and flow of membership:

  • Various scripts in the Scripts & Colabs shared drive folder are hacked-together attempts to pull data from Neon, Airtable, and Sheets.
  • The Class Registration Data Extraction colab pulls event and registration data from Neon and dumps it to CSV for later analysis. This can be useful for detecting occurrence/frequency of particular classes, occupancy and earnings etc. It doesn't include any walk-ins (e.g. techs backfilling not-quite-full classes) which are charged through Square.
  • The Identify paying, active members for a given period colab crawls through all accounts on Neon CRM, accumulates membership information, and uses the data to compute a histogram of new, leaving, returning, and sustained membership on a month-to-month basis. It's not super optimized and takes several minutes to run.
  • Please be careful when executing scripts here, as some of them actually mutate data when you run them.

Materialization of these scripts from previous runs:

  • Membership analysis sheet using the output of the paying/active members colab above, plus a couple of rules and charts to track how we're doing on acquisition

Creating New Metrics

Some questions to answer before creating a new metric or dashboard:

  1. What questions do we always need to answer, on a regular enough basis that it's worth paying (time, attention, effort) to maintain a dashboard/pipeline for? What is this frequency/basis? Is the audience just the M&P committee, the whole board, or broader?
  2. For each of these questions, can it wait a few minutes, or do we need an answer at any moment? (operational metrics vs higher-level reporting)
  3. What would be an ideal “data view” or schema that would support deeper, ad-hoc analysis to answer various questions that emerge from the questions in #1?
software/metrics.txt · Last modified: 2024/06/05 16:47 by smartin015