The importance of establishing a baseline when measuring success

Measuring success is important for both client and service provider to ensure success is achieved.  Everyone has their idea of what success means in the application, so successful delivery and good user engagement might not translate into success in the eyes of the stakeholder.

This is why quantitative metrics need to be established.  Everyone involved agrees on a central set of metrics to which delivery success is measured.  Throughout the project, all success factors are continuously monitored to ensure well-rounded project success.

I’ve already posted an article about the metrics that matter and which ones you should be paying attention to, so I won’t touch on that again here.

Tracking project success is a positive, so you should try to put a positive spin on each metric you’re tracking.  No one wants to see metrics on the negatives (sure, it’s nice to see how a negative has improved, but you don’t want that negative clouding a wonderful metric).  Although valid, these metrics should look and sound like success, while having the numbers to back it up.  Consider:

• “happiness level” rather than “how much less the site sucks” and apply the length of stay as a metric.
• “all the happy shoppers who completed their checkout successfully” rather than “how many people abandoned their shopping experience” and apply a checkout goal as a metric.

Yes / no questions are handy here, but less of a traceable metric and more of a success point (things like “is the interface mobile responsive“, or “does it send email verification now?”  For the project to be a success, all these questions generally need to be answered positively anyway and are expected.  Success is measured in ongoing engagement.

What and how to measure

General metrics I like to gather are actual site metrics, business requirements metrics and those based on end user expectation.  Once measured, try to estimate what success means for each and agree upon realistic successes with stakeholders (e.g. increase in sales – how much is considered success?  3%?  20%?).

Site metrics

Review your current metrics (unique visits, page views, bounce rates, etc.).  These provide rough insight into which areas of your site needs the most work.  Traceable goals here may include:

• Reduced bounce rate (abandons).
• Increased unique visits (more visitors!).
• Increased page view length (engagement).

Get these for the 12 months leading up to project start.

Business requirements

These are generally your primary goals.  The “Why are we doing this in the first place?” questions.  Stakeholders will have a solid set of answers with many possible pain points and goals established.

Traceable goals here may include:

• Increased sales (more money!).
• Increased engagement (which can be measured in longer stays and reduced bounce rate).
• Lighter maintenance (less work for administrators, which can be measured in billed effort, if available).

User experience goals

Stakeholders have their goals in mind, however building an awesome application is ultimately up to its adoption by users.  Decide carefully how you want to measure this.  Consider having ideation sessions with:

• End-users (a set of users who are representative of all who use the site) – this can be via a survey to a large group or in-person sessions with groups, or may be an outcome of UX sessions with end-users.
• Administrators (those who administer the application, whether it’s content or technical).
• Front-line customer support (the people who handle customer service enquiries).

This will result in goals that are very specific to your project.  Traceable goals here may include:

• Users spending less time to get to a goal (shorter conversion length).
• Users spending more time in their visit to the application (to interact with the application or view content).
• Reduction in the number of support requests (phone calls, emails, etc.).
• Reduction in maintenance time due to easier / automated processes (by measuring how much time administrators spend authoring content or making updates)

When to measure

Decide how you want to measure success for each metric.

• I like a 5-point linear scale where actual numbers are not available, e.g. user engagement or time savings.
• Actual numbers speak volumes, so use them where applicable, e.g. support requests, or increased sales.

If you want to compare 2 metrics with each other, use percentages to calculate, e.g.  increased user engagement (which is time-based) and decreased support requests (which is an integer).

Use your tools

Google Analytics has some really nice goal tracking tools, which allow you to see how far people are getting down a particular path before abandoning or completing it.  There are other tools available, like HubSpot which also allows you to review user journeys, specifically those that involve campaigns.

To visually capture and compare these metrics, tools like PowerBI from Microsoft are key.  PowerBI allows you to live-filter through data, automatically filtering other metrics when a single one is selected (e.g. seeing all metrics change when you select people of a certain age as a filter, etc.)

I have my baseline metrics, now what?

You’ve gathered your baseline metrics at the start of your project and chances for success look great!  After the new application becomes available to end-users, run the same exercises again and gather the same metrics as at the start of the project.  Hopefully they’ve increased and decreased where they’re supposed to.  Consider the amount of time necessary for user adoption to not sway results by improper training (where necessary) or infrastructure glitches as part of a large-scale roll-out, and conduct the survey when users are most likely to give you accurate and candid feedback.

• Evaluate with similar paradigms if possible (if your end-users answered a survey, consider sending out a similar survey, asking similar questions but for the new system once users are settled into it).

In Summary

Each project is unique, and your current project is probably nothing like the one you’ve done before.  The metrics you gather are just as unique, you just have to ask the right questions to the right people to get them.  You will be surprised at how successful your next project will be because of transparent and traceable metrics.