Why analyse?
It is evident that there is no point collecting data unless it is analysed and used to compare the outcomes against what was planned and expected. If the outcomes differ from what was expected, the project team may need to make some decisions for changes to the project plan. As the SHA project lead, you will certainly need to be informed and may wish to be involved in any decisions about changes.
Getting the sequence right
Level 1 and Level 2 data should be collected as early as possible (‘at the time of the intervention’) so you can monitor whether you are on the right path to achieving your outcomes.
Level 3 data will be collected when you and the project stakeholders are confident that all stakeholders and project participants have had the opportunity to apply any learning and implement tasks or new performance and/or behavioural changes. Remember that you are looking for evidence of consistent, frequent and sustained changes in performance and behaviour.
Level 4 data will only be collected when the stakeholders and you as the SHA project lead are confident that the planned impact outcomes have been achieved and are sustainable. This is likely to be some time after the level 3 outcomes have been achieved.
If you decide you can and wish to convert your impact outcomes into monetary value (Level 5), this will only occur after you have isolated and adjusted your impact outcomes to take account of other factors that may have caused them to change.
If your project stakeholders wish to calculate a return on investment (ROI) or benefit cost ratio (BCR), they must isolate for the effect of other factors. In other words, they must adjust the impact outcomes to remove the effect of other factors on any improvements made. As the SHA project team lead, you may wish to advise them to get some expert advice in order to do this and/or to seek training in the ROI methodology. As the SHA lead, you should treat any monetary benefit calculation that has been reported to you by a project with great caution. You should check that isolation has taken place and ensure that the project stakeholders document transparently any process they have used to isolate their results and the calculations they have used.
The project full cost data should be forecast and then monitored frequently to ensure adherence to budget and then completed when all actual cost information has been collected.
Making sense of the data
When analysing data keep in mind the following:
Level 1
- Are all stakeholders fully engaged?
- Are they are committed to the actions they are supposed to take?
- Is there any evidence to suggest they will not be able (have the opportunity) to carry out their planned role/implement changes in tasks/performance?
- Is there any information from stakeholders that needs to be analysed and potentially acted on (barriers to implementation?)
Level 2
- Do all stakeholders have the knowledge, skills and attitude required to ensure they are capable of carrying out their role and implement the required tasks and/or improve performance as planned?
- Do the outcomes at level 2 provide you with a clear picture of any gaps in knowledge or skills?
- Do the outcomes at level 2 suggest that a particular group of individuals or stakeholders seem to have exceptional or below expected results?
Once you begin to collect Level 3 data, you will also want to analyse the following
- Are all relevant stakeholders applying any knowledge or skills acquired or using any information provided as part of the project? Is there any knowledge, skills or information resource not being applied or utilised?
- Are all stakeholders carrying out the tasks/actions expected with the frequency and to the standard required?
- Is there any evidence of barriers to implementation?
- Is there any evidence of specific conditions or influences that have enabled application of knowledge and skills and/ implementation of tasks and of improved performance?
Remember the rule – ‘ no data means no evidence of outcome’. So always try and ensure you have the fullest possible picture – ideally from 100% of all participants from each stakeholder group.
Good data analysis requires high levels of competency and a commitment to following good process and remaining fair and objective at all times. Project teams should document the process followed in analysing the outcomes and be transparent at all times. As the SHA lead, you may wish to advise them to take some expert advice and/or training in the ROI methodology.
Steps in data analysis
At the most simple level, the process of data analysis should follow the following steps
- Collect all the outcomes together and collate all quantitative data
- Check for any extreme items or evidence of inaccurate or unsubstantiated data and remove these
- Review all the results together and subject it to ‘enquiry’, based on the questions you were seeking to answer
- Use graphs, charts and other tools which help you ‘see’ the data in different ways
- Qualitative data should be treated with great caution - be very careful not to be drawn to strong or extreme statements or perceptions which may not reflect the majority of views; textual analysis requires a high level of skills
- Document the findings in factual statements e.g. 80% of all respondents rated their satisfaction 4/5
- Use the following statement as an important reference point before drawing conclusions: ‘would an outside observer, given the same results, agree that the findings are fair, objective and transparent?’
- Discuss and agree conclusions with all the relevant stakeholders – consider all possible challenges to these conclusions before finalising them
- Discuss and agree recommendations and document together with all decisions taken
If you are in doubt about the competency within the project team to analyse the data, you are advised to look for some expert help and/or training in data analysis.
Who is responsible for data analysis?
Generally, those responsible for collecting and analysing data at Levels 1 and 2 will be the members of the project team. These same team members are also likely to have a lead role in collecting and analysing Level 3 and 4 data.
SHA project leads may also have a participatory role in the analysis of Level 3 and especially Level 4 data. There may also be experts within the SHA with a particular interest in the Level 3 and 4 data who may also wish to be involved in the data analysis and any decision making resulting from conclusions and recommendations following the data analysis.
SHA project leads who are accountable for a funded project should agree in advance with the project team the data analysis steps that the team will take and how they will document their approach and findings and when this will be submitted to the SHA.
There should also be agreement about any decisions for changes in the project schedule or tasks which may occur as a result of data analysis. These decisions may require agreement from the SHA project team lead if they have serious implications for the project timetable or budget.
