Today's strategic pilots could determine how generative AI unfolds across real estate tomorrow. Its impact will hinge on those who are driving creative experiments now, learning rapidly, and forging new paths. While no technology can replicate human judgement, generative AI may enhance our ability to focus where it matters most: on relationships, strategy, and vision.

The future is rarely obvious until it's the present. Generative AI's potential will emerge through practise, not speculation. The most prudent path forward is to start developing experience through selective pilots. Let the experiments begin. Fortune will favour those shaping what comes next.

It can take years and millions of pounds of investment for a big organisation to get to a place where it can consistently count things. Imagine a company grown by acquisition that has 20 different instances of its CRMs and 30 different operational systems.

But what if someone could look at every piece of unstructured data in your business, one file at a time. And make a decision about how to structure it. That’s now on the table and widely accessible with large language models (LLMs). The models behind products like chatGPT are great at classifying and outputting structured content. We’re not far from a world where your entire disparate corporate data can be finely combed and refined by AI.

Of course, there will be system-level issues, like data not being saved that should be or overwritten rather than iterated. There’s also the ever-present problem of the one person who actually knows what all those columns mean never wrote it down. But AI can help there too. Highlighting where you've got systems destroying your data, or gaps in your knowledge - allowing for a targeted, easy fix.

Maybe, just maybe, our AI doesn't even need to do the heavy lifting of creating a beautifully kept garden of data. Imagine all of your corporate data as a training corpus for a not-quite-so-large language model. You've just built a model that can skip the structured data and torturing numbers out of dashboards, giving you insights about your business from a different route. The insights and opportunities a model like that could surface could be game-changers. It’s not long until the workflow for making insight driven decisions could be totally different.

This is an incredible opportunity for less data-mature businesses that are still on the journey to beautifully governed, structured data. LLMs offer a fast-track route to getting data mature and the opportunity to shortcut the long journey to get to the payoff sooner. As long as your data is digitised (yes, there are still a good number of businesses with physical vaults of data locked away on paper), maybe you can skip the 5-year journey through multiple stages of maturity. Jumping straight to the deep insights, predictions, and actions you could only otherwise get from the long and expensive data maturity journey.

What even is data driven decision making workflow?

There are four steps that generalise an analyst’s work. Question, data, insight and action. They do not make an easy to remember acronym. You won’t see me trying to coin QDIA. Let’s take a look at each one.

Question: Someone, somewhere, has a question! They think that there’s an answer to that question. They start talking to analyst types, or open up the place where the numbers live on their computer and work through the steps themselves.

Data: There’s a lot going on here. Data is shorthand for all kinds of analysis and research. Armed with a question, the next step is finding, creating and analysing the data to try and find some evidence to back up an answer. Often, you need to loop back to the question. You may not have the data you thought, or you may discover something that challenges the validity of the original question.

Insights: Here we have a question and some evidence but we need to form this into a coherent narrative. To check if our reasoning is sound and to communicate. Whoever asked the question needs that answer to spur action. This part often involves a lot of writing. Most analysts don’t like the writing part. Business is still mostly done in words. Like with data you may loop back to do more analysis of your data or back to the original question.

Finally, action: This is the payoff. This is what all the work is for. An insightful and thoughtful answer that empowers decision-makers to do things! Things, because of the evidence and insight provided in response to their well-formed questions, are much more likely to help them achieve their business goals. Increasing revenue, creating better experiences for customers, and growing market share, to name a few.

😱The horror

The problem does not come from little niggles here and there. Every step of the workflow is dysfunctional. It’s a mess. There are the odd outliers where an organisation has gotten this workflow sorted. I love to hear about those. But when you meet people who grew up in the rare company like that, they often talk of the horror of realising what the norm is when they leave.

A pessimist here would argue that any endeavour trying to overcome our innate humanness and bias is going to be a hot mess. I disagree. We can do better, much better. Just because it’s hard doesn’t mean we should shy away from doing better. And doing better can pay off for everyone. I’ve written about the payoff from data-based decision making and the added value of defending against bad decisions and making more good ones. Spotting failure faster and being quicker to adapt to a better course. In a world where that’s the norm, where the insights workflow is a smooth, fast, action-generating machine, products are better, customers are happier, companies are more productive, and everyone’s less annoyed. I want that world. And I think that we can actualise it. We’ve got a long way to go, but it’s within our reach.

Where does it go wrong?

This section may be disturbing. Our careers are littered with particularly awful runs through the insight workflow, and that kind of pain lingers. I hope you can get through it without being too haunted by memories of badly informed action or questions that never lead anywhere.

A word on KPIs

There’s a whole class of numbers, key performance indicators (KPIs) and core business stats that should be generally available. Metrics like ‘how much did we spend last month’ (thank you to the discipline of accounting for sorting that one, mostly). Unfortunately, the push for ‘self-service led by dashboard software providers like Tableau went a bit too far. Yes, your core business metrics can be self-serve, if they’re very clearly defined and stable and your data is in good condition. No, your CEO should not be trying to guess whether the outsized sales performance of your Bracknell branch is down to one particularly spectacular sales rep (get back to making sure there’s enough money in the bank, Ms. CEO).

So for all the really good questions that need some thought, we’ve got analysts to help. Or, if we’re lucky, our own analytics skills too. So many people to guide us through the workflow and make sure we get to action at the end. Oh, what a dreamy world that should be. Were it not for a litany of dysfunctions.

Question

The first version of the question for any meaningful analysis is junk. If you’re just taking a raw question (especially from someone senior) and throwing it at some analysts, you’re doing it wrong. Questions have context and a specificity that never comes out the first time. If it does, it’s obvious enough to be in a weekly or monthly report. Like, ‘how much did we spend last month? If there isn’t some refinement and specificity to a question, the whole workflow is going to go boom💥.

The problem is, if you start with the wrong question or it’s too vague, everything that comes after is wasted time and effort.

Question Fixes

Simply put, don’t waste time on crap questions, but that’s rather hard.

1. Spend more time refining the questions, you can enforce hypotheses, if you want to get very structured

2. Make question refinement a distinct and non-optional part of your workflow

3. Educate the decision makers on what a good question is and why it is

Data

Data and analysis has had the most attention from problem solvers and tech companies. There are a bunch of amazing tools ranging from Excel (yes I said it, Excel is amazing in its own way), notebooks, dashboarding software like Tableau and Looker etc. We’ve got a gazillion databases and ETL tools. It’s hard to keep current with the torrent of data tools. There’s also lots of professional attention in this space, data governance, chief data officers, heads of data etc.

So providing that the question is good, data is more like changeable weather. If you’ve got the right infrastructure (clothing), it’ll be fine most of the time. Until of course you discover you don’t have the data you thought you did, or it’s messier than expected. One thing that amazes me, that despite the amazing array of tools for everything in this space, the majority of analysts still spend a lot of time copying and pasting data from one place to another, or squishing together csvs.

Data Fixes

1. Invest, carefully, in foundational data things. That’s data governance, tooling, access etc, everything you need to make sure you’re data is well documented, clean and easily accessible.

2. Find some of those data professionals I mentioned to help out

Insights

Having done the analysis and worked through the data, now we’ve got to inject some soft skills into the workflow. And this is hard going for many analysts. For most analysts communication, especially writing, isn’t their core strength. To put it in the words of one analyst I recently spoke to, ‘numbers are easy; words are hard.’ If you can’t get your insights into a format your audience can digest, they’re not going to land.

Insight fixes

1. Help your analysts communicate, anyway you can. That might include training, process around how and what to communicate when, AI writing tools and coaching.

2. Keep the feedback flowing, analyst to analyst, decision maker to analyst and so on. Insights should be examined and questioned where there’s time to do so.

Action

Action is a trigger point for me, I’ve spent a lot of time screaming into the void about the failure to link insight to action. It’s why I left corporate life for a start-up five years ago. My career is punctuated by instances where a lapse in political awareness combined with righteousness about insight driven action has nearly gotten me fired. Apparently there is a time and place for contrary facts. I’ve not always gotten that right.

Poorly timed feedback aside, most insight is cursed to never be acted on. That upsets me. If the insights don’t inform some action, what’s the point of all that work? People are wasting their time creating insights no one uses and businesses are burning cash. It costs just as much to generate insights you don’t act on as it does to generate those that lead to great outcomes.

Action fixes

1. Make sure your questions are good (it’s a circle!)

2. Culture plays a big part here, set the expectation that decision makers will seek and use insights to inform their decisions. Make sure it’s clear how you expect them to balance speed vs certainty.

3. Keep a track of the actions that came from insights, make the link to each individual piece of work. If you can’t, there’s a problem to try and solve! How many questions did you answer last month? How many of those lead to action?

Why am I optimistic about the future?

I think we’re on the cusp of a revolution in insight driven action. So much so that I started a business to drive that revolution.

With technology continuing to leap forward, the latest being LLMs like ChatGPT. We now have access to powerful tools and platforms that can streamline the entire insight workflow. From advanced analytics software to AI-powered writing assistants, technology is helping us overcome many of the hurdles that have previously bogged down the process.

As more traditional tasks become automated, our focus can shift to the harder problems of generating valuable insights and making informed decisions based on those insights. There will be more space to put the effort into shifting the culture toward data-driven decision making. As this mindset becomes more prevalent, decision makers will be more inclined to seek out and act upon insights derived from robust analysis.

Alongside this, there’s an underlying hunger across industries to improve how we handle data and use it to inform our decisions. This collective appetite for growth will undoubtedly fuel innovation in both processes and tools. It’s not too long before we’ll be looking back at the current state of insight workflows and wondering how we ever managed without their more advanced successors.

As we continue to embrace technology, foster a data-driven culture, invest in education and training, collaborate more effectively, and strive for improvement, I'm excited about a future with more and better insight driven decisions.

Subscribe now

It's no secret that seniority often influences how feedback is taken. This dynamic can cause confusion and delays in the product development process. Especially if the person giving feedback has founder in their title. An unguarded comment about a feature or design choice can spiral into re-designs, second guessing and shipping delays.

Feedback is a gift, but you can choose whether to accept it

What if I told you there’s a fix?

You can help your team decide whether they should accept and act on your feedback, or file it away for future consideration. Categorise feedback into three groups: must, should, and could. By doing so, the person in charge of the work can make informed decisions about what they have time to incorporate before shipping the product. This categorisation not only helps manage perfectionism but also encourages a culture of learning and continuous improvement.

This feedback model allows senior stakeholders to provide valuable input without hindering progress. They can offer minor suggestions or context for future work without derailing current efforts. Ideally, most feedback will fall under the "could" category, with only critical or deal-breaking concerns being upgraded to should or must. Anything that makes it to must or should needs to be accompanied by a justification for why.

Before your musts, shoulds and coulds though: always start by highlighting the good. By praising what you like about a team member's work, you'll see them deliver more of it in the future.

Next time you're giving feedback on a project or task, keep these points in mind:

1. Begin by emphasising what you appreciate about the work at hand. Taking a moment to recognise your team member's efforts communicates where value is being added and will encourage your team to deliver more of the good stuff.

2. Categorise your critical feedback as must, should, or could. This method helps maintain a sense of urgency and focus while allowing room for continuous improvement. It also ensures that feedback is both constructive and actionable, which ultimately benefits everyone involved.

3. Make this feedback model the norm. By creating a culture where everyone feels empowered to share their thoughts in a structured manner, you'll foster an environment that promotes growth, collaboration, and innovation.

Finding the sweet spot between an open feedback culture and shipping fast may seem like a daunting task. The good news is this structured feedback system that highlights the best aspects of work while categorising critical input will get you a-lot of the way there. This approach will not only help manage perfectionism but also create a learning-focused environment where all team members can thrive. So go ahead and embrace the art of balancing feedback and shipping fast – your team and customers will thank you!

A couple of examples to help you on the journey

Example 1 - slide deck for a pitch to a big potential client

• Good - you’ve tailored our sales narrative really well to this clients specific problems and challenges, it creates a great story.

• Must - The pricing slide is out of date and needs to be updated to reflect our recent changes in pricing strategy

• Should - Take a look at the customer testimonials on slide 12. While they are positive, including more specific details about how our solution has helped their businesses will make a stronger case for our product

• Could - The graph on slide 18 does the job, but I feel like it could be clearer. Could you find a way to show the data without using a double axis.

Example 2 - a task automation feature at the design stage that a CxO is giving feedback on

• Good - The designs for the user interface feel really intuitive, and I believe this feature will greatly benefit our customers by streamlining their processes.

• Must - Before we roll out this feature, we must address an important issue with the date selector. It currently doesn't account for weekends and holidays when calculating deadlines, which could lead to inaccurate estimations for our users.

• Should - I think we should consider adding more customisation options for automation triggers. Giving users the ability to define specific conditions, like task priority or assignee changes, would make this feature even more versatile and valuable.

• Could - If there's time available, we could improve the onboarding experience by including an interactive tutorial or tooltips that guide users through setting up their first automation. This isn't critical but would be helpful in ensuring a smooth adoption of the new feature.

Subscribe now

Another tech hype cycle?

Of course technology hype cycles are familiar. A new technology comes along and captures everyone’s imagination, whipping up a frenzy. The hype builds to a peak, only to come crashing down as the technology fails to live up to unrealistic expectations. The thing is, for those who can see past the hype, the technology will eventually mature and stabilise, leading to real and sustained progress.

Blockchain and cryptocurrencies followed this arc, with hype and speculation reaching peak levels in 2017 before crashing back to reality. I don’t know where to put today's excitement around models like ChatGPT, but it feels early in the cycle. There’s a lot of excitement, and many companies investing in the technology. To me, so far, the opportunity for change and impact justifies that reaction. I’m sure there will be a peak and a crash, likely years away, but after it we’ll find that we’re using generative AI like ChatGPT in our daily lives.

What's different?

The thing with ChatGPT, unlike crypto and blockchain. Is it's already in your business! While those technologies have tended to need a large amount of investment and implementation before payoff. Your colleagues are using chatGPT at work. You just don’t know it yet.

I recently surveyed a small sample of professionals and knowledge workers, finding that 1 in 20 professionals is actively using ChatGPT. That might not sound much, but that's 5% of your workforce using a technology that for practical purposes didn't exist a year ago. Given Microsoft and Google are about to wedge AI language model features into their office docs, this is going to pop up in your productivity tools very soon.

What not to do

I’ve heard stories of some whose knee jerk reaction is to go into defensive mode. Lock out chatGPT from company computers, decree a purchasing ban on any AI powered products, warn employees away, from a working group perhaps. In short go into turtle mode.

The two reasons given for this. First, what if the model is wrong? how do we deal with that? A challenge absolutely, but one we’re used to managing. Has anyone met people? They're wrong all the time.

Second, data protection. This falls into the category of “do not put sensitive commercial data into random free tools on the internet." A perfectly reasonable expectation, but it should be treated like other technology. Train your team not to do stupid things with the data.

The problem with prohibition is that it has zero chance of working. A ban just pushes the use of ChatGTP into peoples personal devices and into secret whispered corners rather than into a place the organisation can understand and build on.

The other extreme. The opposite of prohibition is running full tilt at generative AI trying to squeeze the technology into every corner of your business and product, as quickly as possible. This comes with all of the pitfalls of any other strategy based on ‘I have a solution, where’s a problem I can hammer with it?’

So what can you do?

Maybe you’ve already got a plan. That’s great! If not, first off, you need to get up to speed. But there's a problem. The leaps forward in technology have been so huge, none of us really know where this is going! That means articles written a few months ago may already be out of date.

Your best option is to go find people who are building with generative AI. Who've had some time or so to get under the hood of applying the technology to real world situations. Those people might be in your business already, technologists who've been working on side or hobby projects, they might be founders like me building businesses on top of AI models, or others in your networks. If you can’t find anyone in your circles, the discourse on generative AI on Twitter is worth engaging with, especially with lots of people building with the technology in public.

Don’t put on the breaks, try instead finding out who in your organisation is using tools like ChatGPT and what they’re using them for. Is it adding value? How can you capture that value, are there services or products available that allow you to get the value but with the enterprise level security and data protection you may need?

Subscribe now

I’ve been speaking to lots of very intelligent and experienced people in the insights space in recent weeks. Both analysts creating insights and decision makers using those insights to guide their choices. Early on in my conversations I realised that it’s not as easy to articulate the benefits of insight driven decision making as I assumed.

Isn’t it obvious?

Is data driven decision making good? Gets a quick yes. But the basis of that is more of a general sense of the innate goodness of data driven decision making. The follow up question of why? Is normally much more of a head scratcher. It’s easy to get to ‘it helps you make better decisions’, but then the follow up of, how does that help the business? Is trickier to pin down.

Of course part of the difficulty is different organisations will be working towards different goals. Common goals in the private sector boil down into profit, revenue, growth and market share. Public sector organisations will likely have other targets to focus on. A charity with a mission to reduce homelessness. Or government departments that need to keep the lights on while making improvements in areas directed by ministers.

So what is the value?

We can generalise! We can state that the benefit of data driven decision making is that it increases the likelihood of decision makers taking action that will help the organisation achieve it’s goals.

Problem solved? That’s just half the answer though. How might data-driven decision-making achieve this? And does it work universally every time? Surely there’s no downsides?

I categorise the pros into two areas:

1. Making less bad decisions - stopping decisions that will actively frustrate the organisation achieving its goals.

2. Making more good decisions - decisions that will actively move the organisation towards it’s goals.

In this context bad means: avoidable decisions that will actively hinder progress towards an organisations goals. Think spending a huge amount of money on the worst performing marketing channel in a business focussed on maximising profit.

And good means: decisions that will actively move the organisation towards its goals.

Data driven decision making isn’t a guarantee. When we’re talking about the hundreds and thousands of decisions that get made every day at your organisations, we can’t reduce bad decisions to 0% and good decisions to 100%. Rather, we’re tipping the scales, reducing the number of bad decisions and increasing the number of good ones.

Making less bad decisions

The lowest hanging fruit in adopting a culture of data-driven decision making come from downside protection. Reducing the risk of making objectively bad decisions. These decisions will often come from incomplete data, being totally wrong about some basic facts, or being influenced by vested interests.

Some examples of these avoidable bad decisions are:

1. Promoting and celebrating a sales manager who has increased revenue, but lost significant market share.

2. Hiring a Head of Engineering without any engineering experience.

3. Spending limited marketing budget on poorly performing campaigns, that cost more to convert than other options.

4. Investing time and resource in improving product features nobody uses

5. Spending £4 billion on PPE that can’t be used, because it’s not fit for purpose .

The reason making less bad decisions is lower hanging fruit, is that you’re working in the space of insight driven action where the insights are generally acting to correct the more straightforward set of human biases. Like thinking you know stuff you don’t and an overinflated estimation of your likelihood to be right.

It’s normally quite simple technically to create the insights to avoid these decisions. Often a case of counting, not forecasting or inputing. For example avoiding investing time and resource in a product feature nobody uses, but perhaps the CEO has a soft spot for, could be avoided by looking at usage numbers. Zero users have used this feature in the last month, which is compelling. If data and insights are available you can avoid the work before it happens. If not, you’ll find out no one uses the feature after you’ve spent the time and resource on building it out.

Making more good decisions

Increasing the number of good decisions you make requires a more sophisticated evidence based decision making culture, infrastructure and tools. You’ve covered off protecting yourself from making mistakes that are easily avoidable by measuring, counting and surfacing core metrics at decision points.

To get good decisions you need some kind of test and learn framework in place, a way to experiment to discover new insights which will in turn allow you to make bigger steps towards your business’ goals. As such my examples of good decisions have a little process in them, they’re often more than simple yes or no decisions. You’re normally considering the problem from first principles.

Examples of good evidence based decisions

1. Experimenting with different pricing strategies, which allows you to implement a new pricing structure that increases your revenue and profit.

2. Hiring not only an experienced Head of Engineering, but one who is a great culture fit, has specialist knowledge of value to your company, is passionate about your mission and delivers a huge increase in productivity in your engineering function.

3. Using experimentation and customer acquisition data to identify demand in new target markets, which allow you to grow faster and further.

4. Forecasting risk of a high value customer churning (leaving) allowing your customer success team to intervene early and retain the customer.

5. Implementing a smoking ban in public places, which leads to a reduction in smoking-related illnesses and saves on healthcare costs.

But we need to think about speed

Data driven decision making is a way of working and a culture. But it’s also a descriptor of a tool. One that’s not appropriate for all scenarios. We’ve all heard the term analysis paralysis. A weary decision maker drowning in incomplete data and uncertainties.

Every decision has an element of speed. Some need to be made fast, some can be slower. You can build the infrastructure to have data and insights freely available for fast decision making, which can be a major-asset, but it’s likely that if you need to make quick decisions, you’re going to accept a lower certainty of being right.

This is a totally valid strategy, in many cases you need to achieve pace and create a speed of decision making to deliver insights and progress. Speed of decision making is often undervalued in the data and insights space. Speed needs to be weighed against the need for certainty and how reversible a decision is.

The answer

Data-driven decision making enables organisations achieve their goals by reducing the number of bad decisions and increasing the number of good ones. But there are costs. You need infrastructure, processes, and tools in place to surface insights at the right time. Making sure to recognise when decisions need to be quick versus those that have to be right.

1. What don’t you want? A great way to get your head around a tricky problem is inverting it. Instead of asking how you could increase customer retention, list all the things that would reduce retention. You’ll likely find some ideas you’ve not thought of before.

2. Working backwards - turn the problem around. Where are you trying to get to, or what outcome are you trying to achieve? Now work back from that to where you are now. Thinking through a problem backwards, from the end as well as the default forwards will improve your understanding of it.

3. Write a hypothesis. Just the act of writing down what you think will happen using a structured format - ideally problem, solution, outcome can unstick you and clarify your thinking.

4. Ask a computer - chatGPT and other similar services like Bing Chat, are great for exploring problems. Pose your question, add context and then dig into the response. You can keep asking for more and more suggestions to get some more creative ones.

5. Or go analogue. Take a pen and paper, and set a 15-minute timer. Write continuously about and around the problem you’re trying to solve. See how many 💡 moments pop out as you focus.

• No? Not regularly writing hypotheses? Then you’re not making evidence-based decisions 😱

• Yes to all three? Great, you deserve a cookie 🍪. I bet you could structure those hypotheses even better though.

A key part of the analytics checklist is hypothesis writing, here we go into the detail of why hypotheses’ are super powerful and how to write them well.

So what’s so great about writing a hypothesis?

It’s really easy to be wrong. And there are lots of ways that our minds will play tricks on us, altering reality to make us feel like we’re right. Of course, leading us further astray and to terrible business outcomes. Targeting customers that don’t want or need your product. Spending huge sums on marketing channels that don’t work. Building features that never get used. Scaling up and down the wrong parts of the business.

You can reduce the chance of making all these missteps by adopting hypothesis writing as a core part of your day-to-day workflow.

What is a hypothesis?

The hypothesis is a core part of the scientific method, which has its origins in ancient philosophy from the likes of Aristotle and Socrates. I’ll save you the lecture on the philosophy of science, and just say that the scientific method relies on an observation, which is then investigated, a hypothesis created and then an experiment designed and conducted to test whether that hypothesis is true or not. In a commercial setting we’re generally not applying the rigour of the scientific method, speed is more important than being right every time. So hypothesis writing is a little lighter. The framework I favour for creating a hypothesis is Problem - Solution - Result.

When is a hypothesis useful?

Any experiment you want to run needs a hypothesis. Ranging from the small e.g. testing the effect on conversion of a ‘Get Started’ call to action (CTA) on a landing page vs a ‘Sign Up’ CTA. Or the large like targeting your product towards a whole new customer group. Remember these can be used internally too. If you think redesigning a back office process might save time, write a hypothesis and measure if it does.

The experimentation flow looks like this:

1. Observation - an insight that could come from anywhere. Customer feedback, an observation from an investor or some exploratory data analysis.

2. Writing your hypothesis - here you dig into the observation, look for some evidence to support it and work out how it fits into your company’s goals. That will allow you to design a test to see if the observation is correct.

3. Testing your hypothesis - running the test! This can be a high volume long running A/B test or it could be showing a new button on the website and seeing what customers do (false door) or even showing prototypes to customers in structured UX sessions. I see a lot of people default to A/B testing where they don’t have the volume or time to run a proper test. There are lots of ways to test a hypothesis, think creatively about all the different methods you could use.

4. Reporting on the outcome - once the test is run and data collected, time to compare your results to the outcome expected. Was your hypothesis correct or not? Did you see enough of an impact to say either way? This needs to be worked out and communicated to the rest of your team.

Explaining the Problem - Solution - Result hypothesis framework

This is a great framework for hypothesis writing. Splitting out the three key elements gives excellent clarity and means the framework is simple to adopt.

Problem - a summary of the problem, including evidence of why it’s a problem, quant numbers, qual insights or both.

Solution - an explanation of what you want to change to address the problem, including why you expect this to work.

Result - the outcome you expect, and the specific impact you expect it to have on your metrics.

Your hypothesis will be between 3 sentences or 3 short paragraphs in length. It’s ok if it gets a bit wordy. You can combine all three sections into a single blob of text, but if you’re experimenting with hypothesis writing it’s worthwhile splitting out the three headings. It’s a good way to leave the training wheels on.

Let’s dig more into the example from last week, here’s a model hypothesis:

Problem: The abandonment rate on the sign-up page is 85%, higher than industry benchmarks. Insights from a recent customer survey highlighted the form taking too long to fill in as a barrier to signing up.

Solution: Users are not sufficiently bought into the product to commit the 15 minutes to complete the entire form that early in their journey. We will test splitting the form into two parts, a much shorter sign-up form, and a longer second-stage form to be completed later in the journey.

Result: We expect overall conversion from landing on the homepage to a paid subscription to increase from 3% to 5%, and abandonment rates on the sign-up page should fall from 85% to 75%.

And here’s a bad hypothesis:

Problem: Our sign up form is way too long

Solution: So we’ll make it much shorter

Result: More people will sign up

Let’s go through each of the problem - solution - result sections breaking down what works well from the model example, and explaining why the bad hypothesis doesn’t cut it.

Problem

• ✅ “Potential users are not converting because the sign up form is taking them too long to complete” - description of the problem is clear and concise.

• ✅ “The abandonment rate on the sign up page is 85%, much higher than industry benchmarks” - this quant insight which compares the performance to industry benchmarks provides some evidence that there’s a problem here worth investigating.

• ✅ “Insights from a recent customer survey highlighted the form taking too long to fill in as a barrier to signing up” - bonus! The additional qualitative data adds another proof point specific to our customers. The hypothesis would be fine in many instances with the qual insight alone.

• ❌ The bad example fails to explain any context of the problem. What does too long mean? There’s no evidence to support it being a problem. Length alone is not a problem if there’s no negative impact.

Solution

• ✅ “We will test splitting the form into two parts, a much shorter sign-up form, and a longer second-stage form to be completed later in the journey” - A simple description of the change, that doesn’t need further explanation to understand

• ✅ “Customers will be sufficiently engaged later on to complete the longer second stage form” - The logic for why we think the solution will solve the problem is stated.

• ❌ “So we’ll make it so much shorter” - this isn’t a complete solution, how will it be shorter? Where will the data collection that’s needed happen?

Result

• ✅ “We expect overall conversion from landing on the homepage to a paid subscription to increase from 3% to 5%” - The result is tied to an outcome that will affect business goals, in this case, revenue i.e. conversion to subscription.

• ✅ “Abandonment rates on the sign-up page should fall from 85% to 75%” - This is the most immediate change we want to see, including this metric, which should move quickly and allows us to get some early insights from the experiment.

• ❌ “More people will sign up” - But what if those people who sign up don’t go on to convert to any later stages? Targeting signing up alone isn’t going to drive business metrics like revenue.

Beware of ‘just’, I just want a…..

The most difficult problem for any technical discipline is focussing their limited time and attention on the right problems to solve for the business. You want them to work on the most valuable things. Hypothesis writing helps with this challenge.

Say a Chief Commercial Officer, asks for a list of the most valuable customers. It’s unlikely they just want that list for their own interest. There is an associated problem and solution they will have in mind. A hypothesis forces the thinking and collaboration to articulate these questions, which helps refine the question that needs to be asked and the outcome we’re all aiming for.

So for example a question like ‘Who are our most valuable customers’ could become:

Problem: The churn of our highest-value customers has increased from 15% to 25%. Customer feedback indicates they are frustrated with support levels.

Solution: We want to identify who our highest-value customers are and then target them for extra support from the customer success team

Result: We expect to increase the retention of high-value customers from 75% to 85%

This fully articulated hypothesis allows the team to weigh the work against other priorities. It may also flag other problems, for example here it’s not a stretch to imagine that ‘highest value’ might need defining. And it also creates that all-important feedback loop, if the experiment doesn’t work, we need to follow it up by testing out different solutions!

Wrapping up

Making evidence-based decisions is crucial in ensuring business success. Regularly writing hypotheses before running experiments is a great way to increase the likelihood of making informed decisions. Especially if part of a process using the analytics checklist. It allows you to test assumptions, gather data, and ultimately make informed decisions. By identifying the problem, proposing a solution, and predicting the expected results, you can reduce the risk of making missteps that can result in terrible business outcomes. Using hypothesis writing, you can achieve their goals and make data-driven decisions that drive growth and success.

The basic recipe

Plan for 20 to 30 minutes for 6 to 10 people

1. Draw a 4-box grid on a whiteboard

2. Label each of the boxes with a question

3. Give everyone post-its and pens

4. Give them 10 minutes to draw an answer to each question and stick it on the whiteboard (that’s 4 drawings each)

5. Get the team one by one to talk through their drawings asking the team to guess what the picture meant before explaining it

Flexing format

The exercise works well either in person (with post-its), hybrid or remote. For remote and hybrid use a digital whiteboard like Miro, Figjam or Google Jamboard.

If you’re using a digital whiteboard instead of asking people to draw you can get them to find a picture if you want to speed things up or have some reluctant drawers!

Or to sprinkle your icebreaker with some AI magic get your team to magic up images with the likes of Dall-e of Midjourney.

If you use digital tools it’ll look something like this:

You can do anything you like with those questions

My standard set of four is: A Win - A Fail - What are you looking forward to? What are you grateful for? I’ve found this gives a good mix of encouraging some vulnerability but keeping it light and time efficient.

If you feel like you’ve got some big problems bubbling under the surface you want to talk about you can allocate more time (probably an hour or two) and focus the questions on problems. An adaption of the anxiety party or stinky fish exercises. For that try questions like:

1. Everyone is thinking but not saying

2. I’m still upset about

3. I’m feeling anxious about

4. I just need to say that…

Or you can go the other way and aim for a maximum dose of energy and creativity!

It scales up too

If you’ve got bigger groups, just do one question. Give people a time limit to talk through their drawings.

Or break people into smaller groups and get them to report back any highlights when you all come together.

Subscribe now

There are tonics for these ills. Creating a culture of experimentation, where iterative testing and learning are at the centre of how work gets done, is one of the most powerful. But all too often we do the experiments but don’t capture the value. If the experiments are run, but not measured properly, we don’t get the insights we need. We’re just going through the motions. It’s all too easy to get stuck in endless testing, without the payoff that comes from learning.

Over the thousands of experiments, I’ve run in my career so far, I’ve developed a simple tool that will help you consistently deliver value from your experiments. The Analytics Checklist. A 10-point checklist that will ensure that your experiments are delivering the learnings you need to achieve your goals.

I’m a fan of checklists. A well-tested checklist proves incredibly effective in guiding complex processes consistently and effectively. It seems to work particularly well for anything involving technical experts and multiple different disciplines. If you want to go deeper into the wonders of checklists you can pick up Atul Gawade’s ode to the checklist ‘The Checklist Manifesto’.

Today I’ll introduce The Analytics Checklist and summarise its key elements. In future weeks I’ll be providing more details on the key steps, such as writing hypotheses, measurement plans and common pitfalls to avoid during implementation. You’ll likely default to imagining this sitting alongside a product development process, but it’s applicable to any project or change initiative. As well as product and engineering teams this checklist works really well in marketing, operations and sales.

The Checklist

🔍 Getting Started

[✓] Draft a hypothesis for your experiment

[✓] Get hypothesis signed off by checklist owner

🗺️ Measurement Plan

[✓] Write a measurement plan for the experiment

[✓] Confirm that the preferred solution can be measured with data and engineering teams

[✓] Ensure all technical work required to measure is specified and included in prioritisation

🛫 Launching the Experiment

[✓] Test the measurement works in QA before going live

[✓] Test measurement is working as expected after launch

[✓] Document any new events, UTMs or other reference data, following naming conventions

🛬 Results

[✓] Analyse and test against your hypothesis

[✓] Report back results to your team and document

🔍Getting started

The first part of the checklist deals with the most important part of experiment design, writing a hypothesis, you can go into detail here.

If you’re running experiments in your organisation without writing a hypothesis, you’re doing it wrong. You either need to start writing a hypothesis or ditch experimentation and rely on your gut to get you through.

A hypothesis is an incredible tool for refining the original question, aligning everyone and providing a clear brief for analysis. I favour the problem-solution-result framework for writing a hypothesis. It’s easy to understand and light enough to mean you can still move fast while covering the major bases. I’ll dig into the detail of hypothesis writing in part II.

Once you have your hypothesis drafted you need to share it, so in the checklist, we have a sign-off stage. This sign-off should prompt wider communication and a back-and-forth to refine the hypothesis. I’ve found using this checklist that simply including a sign-off step, will mean that wider communication happens too.

🗺️Next comes the Measurement Plan

The measurement plan sets out how you’re going to conduct the experiment to test your hypothesis. The shape of your measurement plan will be different for each organisation, or even team. You may want to add or take away. Generally, you’d expect the amount of depth and detail in the measurement plan to scale with the size of the experiment. The bigger and more complex the experiment the more detailed plan will be needed.

This is lightweight measurement plan which serves as a good starting point:

Hypothesis: problem - solution - result

How will you measure this: Description of the method for the experiment

Can the solution be measured with existing tools: Yes/No

How long will the solution need to be measured: Days, weeks, or months?

Who will I test this on and what is my control group: This could be comparing a vs b in an a/b test, a previous period of time or some other split.

Is there anything else planned which might distort the measurement: The answer to this is often yes, so how are you going to deal with that?

The measurement plan will often surface extra work that might not have been obvious when initially planning the experiment. I’ve seen many experiments launch to discover they’re actually unmeasurable without additional development work. This step is very important because it catches a lot of problems that would otherwise pop up later on, often when the experiment has run for a week or two and no one can make sense of the results.

🛫Time to launch the experiment

Test, test and test! Measurement should be tested explicitly as part of your QA process. Then, again once live. You want to avoid spotting an error or bug with an experiment weeks in, that means you need to reset your clock to get to insights.

And documentation, often forgotten, without a prompt. Somewhere you’ll likely be keeping a record of your data. If you’re creating new fields you need to document those and make sure you follow standard naming conventions. This is particularly important when working with any experiment using UTMs, as chaotic UTM naming will quickly turn your marketing and product analytics into mush.

🛬Wrapping up Results

The final two steps are where all this hard work pays off. Your results can be compared against the original hypothesis and any additional insights unearthed. Then those results need to be communicated and documented, so decisions can be made and actions are taken.

Roles

A final word on roles and responsibilities for your analytics checklist. This will vary according to the organisation but broadly there are two key roles I want to highlight.

A product or project manager needs to be responsible for each experiment and completing the checklist. They should be doing the work, particularly communicating. A senior stakeholder should have overall responsibility for the analytics checklist and provide support and sign off as required. This generally would suit a senior Data or Product leader, but could be anyone with a nose for insights.

Template

And there’s a template in Notion that you can copy for your own analytics checklist here!

I’m sharing five ways to fail at being wrong productively. You’ll likely recognise them all, some maybe in yourself! Remember all of the below exist on a scale, there’s a sweet spot to hit! Are you too far one way or another? Perhaps you can correct for it?

1. Being too afraid of being seen as wrong. A bitter irony. The more you fear being wrong, the more often you will be. Of course there’s a healthy dose of fear of failure that’s motivating, that can improve the quality of your work and the depth of your insights. But it’s all too easy for some to conflate success with always being right. A cure? Try volunteering when you don’t know or inviting feedback and challenge? Take a look around are you creating an environment where it’s ok to be wrong, and learn, can you lead by example?

2. Obsession with absolute truth. Or analysis paralysis. There’s never enough evidence or insight to make a decision, best to commission more research and more reasoning and leave it for another day. But timing matters, often you need to make decisions on incomplete information. It’s rare you can’t course correct if you take a wrong turn. A neat trick is to tell yourself you’ll make the decision, but check in on it in a month, it’s not forever right.

3. Too much science. The only way to find truth is a statistically significant analysis of a carefully designed experiment, run for long enough to accumulate a large enough sample (likely months) to be certain of the answer. Anything less can’t be trusted. A horror someone should highlight a conversation with a customer or a gut feel. You may be a perfectionist with a maths degree! If this is you, seek out more stories, listen openly and kindly to feedback from customers and customer facing teams. Ask what’s good enough for the problem at hand?

4. All stories and no numbers. A single customer sharing a catchy sound bite becomes the product strategy. A truth everything else needs to fit to. Never mind that it’s a sample of 1. And the long suffering product analyst in the corner can’t seem to find a smidgen of data to support the assertion. Leave space for those around you to test your stories, find a data obsessed friend and ask them to take a look. Or learn how to yourself.

5. History re-write. We all do this to a degree. A psychologist would call it hindsight bias. In its worst form it means re-writing the past to always have been right. The problem with always haven been right in the past, is you have no feedback loop, how can you learn? Imagine a leader commenting how obvious a new initiative was going to fail, that they earlier were passionately arguing to prioritise. Try keeping a decision log, summarising what you thought at key decision points reviewing it from time to time, it’ll keep you honest.

Accepting that many of our ideas won’t work is one of product management’s superpowers. Testing and validating our ideas before we’ve committed too deeply to them avoids costly mistakes and leads to spending limited time and effort on the best bets. And we learn along the way.

The thing is an intelligent and experienced person, making reasonable assumptions can be wholly and wildly wrong.

Take an example from an early prototype of a property sourcing system. We wanted to find the best homes from the millions advertised for sale on the property portals. So we decided to filter out homes that didn’t meet the bar for our customers.

Feedback from the team testing the output came back that there were too many caravans popping up (yes you can find semi-permanent mobile homes on Zoopla, like this). The solution was to search for keywords like ‘caravan’ or ‘mobile home’ in the description of every property and remove anything that got flagged. The examples already flagged were used to test the logic. All seemed well.

Here’s a simplified view of how the logic was expected to work

When we came to turn the prototype into V1, we turned a more critical eye to this logic. With entertaining results!

What proportion of homes with the word ‘caravan’ in the description do you think were caravans or semi-permanent mobile homes?

a) all of them were caravans

b) half of them were caravans

c) none of them were caravans

The answer is c. Almost none were caravans (other than the very small number of examples that had been flagged already). Most of the homes were good-sized family homes with drives, roomy enough to park a caravan in. Exactly the kind of home our customers were looking for.

Here are a couple of examples straight from Zoopla:

What about the filter aiming to remove houses of multiple occupancy, HMOs? These are homes with 3 or more rooms let out to individuals like students. Did we remove ‘HMO’s or homes with a ‘fisHMOnger’ on the local high street?

When we did our first pass, very much in a rush to pull everything together. We were totally wrong. With time to take a proper look the second time around we spotted the problems and corrected the error. Implementing a more sophisticated property filtering system, with test coverage!

I particularly like this example as it lacks much of the ambiguity we actually have to deal with day to day. We had proven that the more properties of good quality we showed to customers, the more likely they were to convert. So filtering out properties that fit that bill was bad for our goal of converting more customers.

But would I have wanted us to spend more time developing the prototype to guarantee that the filters were correct? Absolutely not. At the time we needed to ship something quickly that we could test with customers. Yes, it removed properties that were good, but only a small proportion and we had a customer journey flow to ‘save’ any overzealously rejected homes. We made that common trade of accumulating a little technical debt for speed of delivery.

The key then is having a culture that

1. Acknowledges we’re going to be wrong, a lot, and encourages a test-and-learn approach. By embracing the idea that failure is part of the process, teams are encouraged to test their assumptions and ideas before committing too deeply to them. This leads to better decision-making and avoids costly mistakes in the long run.

2. Leave time to revisit work and fix technical debt. In the rush to deliver, it is easy to overlook potential errors and overlook areas for improvement. But having the time to reflect and make necessary corrections can improve the overall quality of the product and ensure it meets the needs of customers. Not to mention allow you to build faster in future.

So, what unknown horrors await in your systems? Do you have a culture where your team can surface those problems? Is there space to tidy up some of the systems left in a mess in pursuit of speed?

Subscribe now

1. The rule of six from An Elegant Puzzle by Will Larson. A simple but very useful tool for thinking about how to design and build your organisation. Each manager should have 6 managers or ICs reporting into them. It’s amazing how, on paper at least, enforcing that rule would immediately flatten your organisation. In practice of course, it gets a little more complex, but I found it a wonderful thought experiment. You can read about it here, the focus is on engineering teams but you can apply the rule more widely.

2. Ask-tell-ask from Being Mortal by Atul Gawande. A model for conversations popular in Healthcare. I’ve been using it to make sure I’ve explained technical problems and concepts properly. Step 1: Ask what the person already knows? Step 2: Tell them what you know or think. Step 3 (the critical one): Ask them what they heard? Following that model, both of you will leave the conversation having understood each other clearly!

3. Writing a press release to explore a new product idea from Working backwards by Colin Bryar and Bill Carr. I first came across this in Inspired by Marty Cagen, but it had fallen out of my toolbox until I was reminded of it. Writing a fictional press release backed up by extensive FAQs is a great tool for articulating and refining product ideas. Here’s a template you can use.

4. Break points in growing businesses based on team size from Build by Tony Fadell. Having been through this and had to re-design organisation wide ways of working multiple times this really resonated. Tony flags the break points as 15 people, 40 to 50 people, 120 to 140 people. Similar to the rule of thumb of every time you double the size of the team everything changes. I think the exact numbers are different for everyone, but you can avoid a lot of pain and a big slowdown if you have something on the team looking out for those break points and planning ahead.

5. Meditate, daily from The Headspace Guide to Meditation and Mindfulness by Andy Puddicombe. I’m a fan of Headspace - a mediation app. It helped me build up a daily meditation practice, which has helped me manage stress, build resilience and generally be happier day to day. Andy’s stories of life as a rogue monk are wonderful. They even made a Netflix series.

Subscribe now

Let’s start with some background numbers, the government employer skills survey (sadly dated 2019, updating coming later this year) is the latest source of info. I was struck that only two thirds of employers had arranged any training for any of their employees in the last year. It turns out, that it’s mostly smaller businesses that haven’t invested in training. 92% of companies with 25 or more employees had run training in the last year. But the budgets were higher than I expected, those that were trained had an average of £2,540 spent on them. Making that typical discretional £2,000 start-up L&D budget look less generous.

The training budget is a particularly popular feature in growing businesses because it’s a cheap benefit to offer, and reads well on a job ad. It’s probably the number one question from candidates about benefits I’ve found in interviews. Although they should really be asking about pension contributions!

There’s a little secret about these types of training budgets. They rarely get spent. In an early stage company only 3 in 10 in your team are likely to spend their training budget. The rest, well, never get around to it. That makes the benefit cheap, a headline of £2,000 per person per year will likely only cost £600, vs say an extra £2,000 salary, which will be 100% spent, every year. Of course, you get what you pay for though.

It turns out the most likely outcome of ‘hey, here’s some money, go find something to spend it on, your choice’ is not lots of learning and developing. It’s another thing for your team to feel guilty about while remaining resolutely at the bottom of the to do list.

Time is a major factor here, everyone’s busy. There’s not enough time to do the day job, let alone extras. This isn’t about performance. It’s just as likely that the most hard working, driven and productive in the team aren’t spending that budget as anyone else. For others it may not be their learning style, they may prefer to tinker alone or read.

I’ve nipped at this problem, but it’s not easy to solve in a start-up context. Some things I’ve tried with mixed and generally luke warm results:

1. Making it a standing agenda item in 1-1s with direct reports, basically nagging the people you work with

2. Using a career development framework to highlight areas for training

3. Finding specific training courses and recommending them

4. Talking about the training alot - in all-hands meetings, company docs, team meetings

5. Having an very lightweight sign off process and defaulting to saying yes

6. Making it clear that time off to do the training is part of the offer too

7. Buying any technical/business books on an Amazon company account, on the proviso you share it when you’ve finished it. This is actually work quite well, for a time, although not everyone remembered the sharing bit!

What are we to do?

For a start, be deliberate about why the budget exists and what you want to achieve as a result. This is the extremely obvious step I’ve missed in the past. Without the why, do we even want 100% utilisation of that budget? What’s it there for? It gets harder and hard to spend that budget, it’s a useful line in the budget when money is getting tight. Off-setting say an overspend in marketing with an underspend in your training budget seems like an easy decision to make. Especially when you’re not entirely sure exactly how to get that training spent, or what it’s really there for in the first place.

For the minority that do spend the budget, they tend to get alot of value. You may have someone who has a very specific career path or technical discipline in mind, so is likely pushing you to spend more than their individual budget to do a bigger course. I’ve generally approved those kinds of requests in the past, and every time they’ve paid off both for the company and the person doing the training.

Another angle on this problem is that the fully honest offer of a personal training budget can be described something like:

”We don’t have the time or resource to provide a structured training environment for you. We probably don’t even know what skills you need to work on, or how you’d work on them. So you’ve got to take care of your own professional development. We’ll throw you a bone. Here’s a budget.”

Like many things early stage, it’s do it yourself, or it won’t get done.

The system level solution is to start by identifying what the goal is, what are you trying to achieve? The use the why to work out a plan, likely including details of career progression, skills for different levels, your teams current individual skills and the routes and training that will allow them to progress. Yes, your Head of People is going to need a larger budget!

If you’re not there yet though, there are a few ways to make that fixed budget without too much support go further:

1. Don’t skip working out the why. Start there. What’s the point of the training budget, how does it fit into the company vision and mission? What are you actually trying to achieve? How does your investment pay off in an ideal scenario?

2. Use that why to explain why it’s there, remind people about it and get your managers regularly talking to their teams about it. With a link back to how the mission and company benefits your team will engage with training more.

3. Measure it! No surprise that I’m advocating for measurement. But if the why is important, you need to measure it to improve it. If you’re brave set a target for how much should get spent.

4. Then role model, expect your leaders to model good behaviour and spend most of their training budget, while broadcasting the fact at every opportunity. As you’re measuring it now everyone will be able to see how well the leadership team are making use of the training budget.

5. Allow and dare I say, encourage, the minority that want to do more bigger things and get them to share their experience. Although do make sure there’s a link back to the why and some business case.

6. Consider taking a chunk of that budget and allocating it to companywide training initiatives in service of the why. There’s lots of platforms you can sign up to help here. Think codecademy or Coursera for business

7. What about a monthly budget? Think about splitting some of that annual budget into a monthly £s cost. For example you could make £50 per month available for subscriptions to training related courses or buying books. £50 worth of training a month is a much more manageable chunk than a £2,000 annual course!

Here are the 5:

1. “…..” Say nothing. Yes, many people hate this. But lean into the silence. Sometimes the best way to find out more is just to leave space. Leave some time, see if it gets filled? Maybe you’ll learn something new. Just be mindful of the rare person who can sit in silence longer than you can bear.

2. “What other options have you explored?” Whenever I can’t get my head around a proposal or suspect there’s a lack of depth to someone’s work I’ll reach for this question. Sometimes the response is ‘none’. Other times looking at the alternatives can better frame the problem space and solutions. Sometimes it’s about picking the least bad option.

3. “Is there anything I could do more of? or less of?” If I feel like feedback isn’t flowing freely, I’ll add this as a standard prompt at the end of any 1-1s. As it becomes expected and people get used to it you’ll find you get feedback more regularly.

4. “If you could change one thing to make this blog better, what would it be?” If you’re struggling to get feedback, this prompt can often get people thinking and talking.

5. “Shall we cancel this project?” This one is from Chris Voss’ book ‘Never Split the Difference’. A project stalled, supplier or colleague ignoring your emails? Want to jump to the top of their priority list? This will do it. Not one to be overused. But this question will almost always generate an immediate response. If it doesn’t, at least you know for sure you’re not going anywhere soon.

What are your favourite questions?

I want to focus on the question, "What have we learned?" Asked earnestly, not in the chiding voice of a grumpy teacher. Tone matters!

Originally I started asking this question more often as a build on our board deck. We wanted to demonstrate the rapid progress we were making as we launched our product, as well as keeping our board’s mental model of our customers and product up to date.

We’d present a ‘What have we learnt?’ slide as part of the board pack, every few weeks. It became a highlight, often triggering debate and follow-up. Sharing a learning about our relative channel performance, triggered a deep dive which lead to us optimising and improving our marketing mix. Insights about our early adopter market showed the progress and depth of understanding we were accumulating pointing to how our product would develop and where we would invest in future.

Of course, early days that slide triggered a personal scramble as I poured through my notes and frantically asked everyone about what they’d learnt, way too close to the time we sent out the deck. But the question quickly became a wider feature of the business. Serving two functions.

First An opportunity to reflect, acknowledge the progress made and then share and reinforce learnings. It’s all too easy to rush forward, never leaving time to review the path you’ve travelled. Learnings end up silo’d, with the team’s mental model of the customer and product out of date or incomplete.

Second, regularly asking ‘What have we learnt?’ is a clear signal, that we expect to be learning. Bringing an idea to market, finding your first early adopters and growing, is a journey of learning. You need to be constantly testing, iterating, learning and improving. The better you are at this cycle the further you’ll go.

Learning is an important feature in most company cultures, in some shape or form. Embedding the question of what we’ve learnt is a practical and effective way to make it an actual feature of your team’s behaviour.

I’ve tried plenty of different ways to encourage experimentation, insight and sharing of learnings, in many different settings. But it’s all too easy to end up with a couple of hundred slides full to the brim of profound insights no one has ever read. Sure sometimes you need a central repository of all your learnings, or a beautifully crafted paper explaining a recent discovery alonside exquisitely detailed tracking of all of your experiments. All of those tools have a time and a place but aren’t as universal as making ‘What have we learnt?’ a regular question.

So if you feel like you need to dial up the learning part of your culture. Here’s how I’d suggest deploying ‘What have we learnt?’:

• If you’re responsible for the board deck. Add a learnings slide to your board deck. Commit to making it a high value feature of the deck - not just filler. We all know not all board slides are made equal, some get considerably less love and care than others!

• Share this slide at company all hands, early on in the session before people’s minds start drifting. Extra points for getting the person responsible for each learning to share the headlines.

• Make it a standard agenda point in your monthly (or weekly) management meetings. ‘What have we learnt?’ Make time for it. Early days come prepared to prompt your colleagues, ask them about things you’ve overheard or anything they’ve been working on. Don’t just expect your product or marketing people to do the heavy lifting, everyone should contribute.

• Add it as a standard item in your 1-to-1s, especially with anyone outside your direct reporting line.

• Then exercise some patience, see if the question propagates. If you’ve been consistent and modelled the question well, it’ll become a standard part of your team’s conversations. You’ll see it on team agendas, being weaved into async updates and appearing as a feature in public praise.

  Of course you can also do it the other way round. Start asking it in your team meetings and see how long it takes for a learnings slide to appear at all hands and the board deck.

Tennis grandslam winners from the FT. Similar to the rainfall table from my post about time and rain. You can go a long way with a table and some shaded cells! This one immediately shows you just how much Roger Federer, Rafael Nadal and Novak Djokovic have dominated men’s Grand Slam tournaments. And also how much a clay court (as in the French Open) seems to favour Rafael Nadal.

3. For an up to date visualisation of UK government spending. Information is beautiful’s income and outgoings graph gives the current picture for the UK. Although I do miss the Guardian’s circles!

   

4. That’s enough of all those colours and complex shapes. What about visualising data for trading meetings? I’m a fan of this graph for showing multiple time periods at once. A visualisation from Amazon’s business review reporting. The graph comes from the book Working Backwards by Colin Bryar and Bill Carr. Visualising time series data, particularly for key business metrics like sales, visitors, sign-ups etc. is a surprisingly thorny problem. It’s hard to avoid relying on pages and pages of tables. Often popular with the CFO, but totally inscrutable for most of the rest of the management team. I feel like if your reports require a ruler to be read, something has gone wrong in the design phase. In this example, the lack of labels on either axis does make me twitch though.

   

5. John Doerr’s Speed and Scale OKR tracker - Creative use of OKRs here, to express huge goals. Save the World! The tracker has a really clear and clean framework for showing the status of the individual key results. I must admit, my OKRs never looked this good!

Subscribe now

Weather data from the met office is more convincing. Looking through the lens of days when it rained, London seems very wet. 174 days when it rained in 2021. If you ran weekly but wrote off rainy days, that’d mean cancelling a run due to rain every other week. Zoom into hours, though and it’s 951 hours it rained in 2021, about 10% of the time. You go from skipping a run every other week to skipping a run once every two and a half months.

Here’s the number of hours it rained every day in 2021. The deeper the blue the more hours of rain.

If you don’t think about rainy days, but look hour by hour, you’ll find lots of time to go for a run, walk or swim. Failing that there’s always the wisdom of Alfred Wainwright ‘There’s no such thing as bad weather, only unsuitable clothing’.

The conversation struck me as an example of two important concepts and pitfalls when trying to make decisions from data.

1. Zooming in and out, or adjusting granularity, can unearth new insights. As with rainfall.

2. And that time is often tricky to work with, a cause of many errors and misinterpretations.

To illustrate a pitfall. A new member of the security team raises an alarm over a spike in traffic. There’s been a 10x increase in traffic in the last 2 hours, much bigger than anything else on the graph! Are we under attack?

Time to zoom out, how long is the date axis on that graph? In this instance a rolling 3 days. There’s a monthly and weekly cadence to our marketing activities, how does it look over a full week or month, compared to last year?

Zooming out shows the spike is a normal part of the marketing rhythm and nothing to be worried about. Everyone can stand down.

So what are we to do?

When you’re looking at analysis or reporting you need to have four questions at the ready:

• Time frame: What time period does the data cover?

• Units: What granularity of time is being used? minutes, hours, days, months?

• Context: Are the time period and units right for the question I’ve asked?

• Sample size: Is there enough data to rely on the answer?

Any time I hear a number, my first thought is ‘When?’ Sales grew by 50%, between when and when? Days, weeks months? This is a radically different stat if it’s between 2021 and 2022 vs 11.00 this morning and 13:00 in the afternoon.

Sample size factors in here too, as you shorten the period your data covers you’re reducing the sample size. That means more volatility and more likely you’re going to take the wrong read from the data. I should caution. In my experience, most people asking questions of their data want to do so at a granularity that’s too volatile to deliver useful insights. This can lead to bad decisions and wasted time. Of course, there are lots of statistics we can use to try and work out if the movements in our data are just noise or meaningful. But you can get through most day-to-day challenges with a simple rule of thumb. If your sample is less than 50 for each data point, take extra care!

So why not get outside? There’s only a 10% chance it’s raining.

A few quirks of date and time that catch us out

• In America the standard date format is month/day/year, in Europe, it is day/month/year. Is 12/01/2023 the 12th of January or the 1st of December?

• 24 hours in a day, 60 minutes in an hour, 60 seconds in a minute, 1000 milliseconds in a second.

• Leap years have an extra day.

• Time zones and seasons, need I say more?

• Daylight saving time - the time is an hour different according to the season in some countries.

My own reading journey has had many ups and downs. While I spent most of my teenage years with my nose in a book, I seemed to lose the habit and practice of reading in my 20s and early 30s. In more recent years, I’ve found a steady rhythm. Helped by a few simple routines reading books is a regular feature of my day to day life. I find reading immensely rewarding, but there is friction to overcome to read regularly. You need to make time and space, and books can take time to warm up to. You don’t quite get the instant gratification that you might sitting down to watch Netflix or listen to a podcast.

Reading books has a lot of social status attached to it. It’s easy to lose sight of why you’re even struggling through that incredibly important popular anthropology book that everyone told you to read (yes, I hated Sapiens and abandoned it halfway through). There’s an intellectual pressure to read deeply and widely, that we all feel to some degree. I suspect though that most of us read fewer books than you think.

How many books did you read last year and how does that compare to everyone else?

A Pew Research Center report puts the average at 12 books a year for adult Americans. That’s skewed by a small number of people that read loads. The median is 4. So we can keep 4 as the typical number in our heads.

Surveys from YouGov let us look a little deeper into that number for UK adult readers. I read 32 last year, which puts me in the 88th percentile, i.e. in the top 12% of readers by volume. To get into the top 10% it’s 30 or more books, top 5% 65 or more.

Here’s what it looks like if we had a hundred people lined up by order of how many books they read in a year. If we walk alongside this fictional queue, we’ll find the first 17 people in the line haven’t read any books. Somewhere around the middle, we have 17 people who’ve read between 3 and 5 books. At 30ish books, I’ll be number 90, with 10 people ahead of me. Then right at the front, those power readers, a small group of 4 who’ve read 71 or more books in a year!

Enter the book of books

The way get to the front of that queue and read more, is to develop a set of habits and practices around reading. My favourite and most effective reading ritual is a Book of Books. A notebook where I keep a record of everything I’ve read. There are just two key elements to the notebook.

1. A list of everything I’ve read for any given year (you’ll notice I like to score books from 1 to 5 too)

2. A short write-up of each book after I’ve read it

Here’s my reading list from 2019/20.

And a write-up of my favourite book that year ‘The Phoenix Project’ by Gene Kim (typed up so it’s legible).

The Phoenix Project

Gene Kim, Kevin Behr and George Spafford

11/2019

5 out of 5

This is the best business book I’ve ever read. It’s about devops, agile and technology in ‘real’ organisations. It’s written as a story (in the style of The Goal), following Bill IT (the horror!) manager at Parts Unlimited. All the senior team are fired and Bill is put in charge.

The business is failing and only their ‘Pheonix Project’ can save it. Through his journey, Bill learns the foundational principles of Agile and applies them to create a successful devops organisation.

The first thing that struck me is how much of a well established pattern some of the bigger corporates I’ve worked in fit into. No one leading those organisations seems to be aware of those patterns or thinking about trying to avoid them.

This book really connected the dots for me on the theory of constraints and agile methods generally. It’s helped me re-frame business problems and I’ve got follow up reading planned in.

The 3 ways

• The principle of flow

• The principle of feedback

• The principles of continuous learning and experimental

General concepts/to look into

• Prioritise running tasks to completion, not batching them together

• Look into - High trust management cultures

• Resilience engineering

• Applying stress and pressure to a system to ID failure points

• Plan - do - check - act

• 3 acid IT death spiral

• Categories of waste

• Swarming

How does this help with reading more?

There are a few problems that the Book of Books solves and habits it encourages.

Recall: I can entirely forget books I’ve read, to the point where I’ve found myself halfway through a spy thriller and realised why it all seems so familiar. No, it’s not just a particularly formulaic Jack Reacher book, I’ve read this one already! By some trick of neurological magic though, if I write a few sentences about that book around the time I read it, most of it will come back to me.

Depth of understanding: Reflecting on a book and writing up your thoughts and its influence gives you a greater appreciation for the book. It helps place it relative to other literature and I often find you can draw out some additional insights or pleasure as you write.

Measurement: If you do want to read more, keeping an honest account of what you’ve read is going to help you. I started my own book of books in 2017. Since then I’ve gone from 15 in 2018 to 32 books in 2022.

It’s easier to talk about books with friends: Ask me for a book recommendation and I’ll reach for the book. Likewise, you’re reading something I read a few years ago, a quick refresher and I’m ready for a meaningful chat about the contents.

Bonus, decluttering: I’m much more relaxed about passing on books now. I used to worry if I didn’t have the physical book anymore, I’d forget what I’d read from it. I used my many bookshelves as a record of books past read. Which of course led to full bookshelves and random piles scattered throughout the house. Now I can keep a core set of favourite or sentimental books and they don’t overflow the bookshelves. Freedom from book clutter!

A recipe for your own book of books

Start with a brand new pristine paper notebook or a space in whatever app you use for your digital note-taking. For example a table in Notion or a notebook in Evernote.

1. Write the year at the top of a page.

2. Read a book

3. Write the title and author of the book on the page, with the date you read it

4. Give the book a score, and write that down too

5. Write a page about the book you read. What did you think about it? This is a really good reflection point, I often take a look at what some other people thought of the book and leave it a couple of days to let everything set in.

6. Repeat steps 3 to 6

A few extra tips and tricks you might want to consider

Reading is of course a habit, find a time that suits you for reading and try and do it every day. Perhaps half an hour before bedtime? 15 pages first thing in the morning?Lunch with a book during the week?

Don’t punish yourself by finishing books you hate. If you’re not enjoying a book put it down, if you never pick it up again, what’s the problem? If you really want to read more you’re trying to foster a love of reading, find the books that fuel that passion. Ditch anything else.

Find some other people reading books. Talk to them about reading, get recommendations, debate what you loved and hated. Bookclubs pop up everywhere too, ask the people around you whether they know of any. Or want to start one!

If you’re struggling with what to read ask those that know you best to buy you books when they’re buying you presents. Don’t be too specific with genres or types, encourage them to share what they love or what they think you’ll love. Many of my favourite reads come from friends and family. Whenever I get a book from friends or family it jumps to the top of my reading pile.

A word on quality vs quantity

Measuring and setting goals related to the number of books you read is an excellent example of the perils of measurement and over optimising for KPIs. Imagine for a moment I’m paying you to read 50 books in a year, with no quality measure in place the rational choice would be to find the shortest and easiest to read books smashing through your target in a few weeks. Had I made a business of paying people to read books (stay with me) and we were writing OKRs for the team we’d probably try to prevent the gaming of the system. A key result draft might look something like ‘Read 50 books, of at least 250 pages each in 2023’ or ‘Read 50 books from the list of the most 1,000 important books ever written.’ But How are we to weigh the value of a book that’s 1,000 pages of dense text vs one that’s 100 pages? Or a book that’s popularly considered important, vs something that very specifically talks to you and your own lived experience?

The good news is, I’m not paying you to read. So you don’t need to game the system. Setting a goal is entirely optional. You can choose to read fewer longer books. To flip between short fun reads and long intellectually enriching classics - or just completely ignore book length. Some books can deliver 1,000 pages worth of insight in 50. Others can offer none in 500.

What about podcasts, articles and even blog posts

I don’t think there’s much need to pit different formats against each other. They all have their place. And that is a very personal relationship. For me, there’s something about a book, in terms of length and intensity, that sets it apart from many other sources of content. Mix and match how you like. It’s your time and your attention.

Leave a comment