Ukraine Conflict Monitor - Accountability and credibility

Assessing our analytical performance in 2023

This week’s post will be different as there is something we would like to share with you.

As you know, we pay particular attention to accountability. We weigh our words and strive to provide impartial and correct analysis. Each weekly post includes a forecast for the upcoming seven days. There, we also reflect on how we did in our previous assessments. So far, our scores have been quite high. In part, this reflects on the current character of the war, when neither side can achieve a breakthrough leading to faster gains.

But, apart from the dedicated forecast section, we also make predictions in the Executive Summary section. These assessments are more difficult to track and evaluate as this requires a concentrated effort. As these statements are not bulleted, they can also be often lost in the mix of everything we publish.

Today’s post will be about assessments published in the Executive Summary section 2023. I asked my colleague to take a hard look at what we wrote there over the past 12 months and attempt to evaluate our posts from a customer perspective.

There are several reasons why we decided to undertake such a task. From our point of view, we seek:

1.      Continuous Improvement: Such assessments allow us to identify areas for improvement in methodologies, sources, and analytical techniques. This iterative process helps refine and enhance the overall quality of our intelligence products.

2.      Adaptability: Assessments facilitate the identification of strengths and weaknesses in analytical approaches. This knowledge enables us to adapt strategies, methodologies, and tools to address emerging changes on the battlefield or within Russia or Ukraine.

3.      Accuracy and Reliability: Regular self-assessment ensures the accuracy and reliability of intelligence assessments. It allows us to validate the information, verify sources, and correct any inaccuracies, contributing to the credibility of the intelligence produced.

4.      Risk Mitigation: Assessments help identify potential biases, assumptions, or gaps in intelligence analysis. Addressing these issues mitigates the risk of making flawed decisions based on incomplete or inaccurate information.

5.      Building Trust: Regularly evaluating and improving analysis builds trust with clients and stakeholders. Demonstrating a commitment to accuracy and transparency enhances the credibility of the intelligence agency or organisation.

From your point of view, we realise that you rely on accurate intelligence for informed decision-making and effective risk management. Or better news and analysis sources. Accountability in intelligence analysis ensures that the information provided aligns with your needs, enabling you to make decisions based on reliable, credible insights or be simply better informed. Customers benefit from the transparency and feedback loop established through regular assessments, allowing you to trust intelligence providers (us). Ultimately, accountability should enhance the overall confidence of customers in the reliability and integrity of information received.

In summary, regular assessment and accountability in intelligence analysis are essential for maintaining accuracy, reliability, and credibility. It enables organisations to adapt to changing circumstances, make informed decisions, and build customer trust.

Going back to our work. Altogether, we assessed 146 claims, of which:

• 103 were correct;

• 15 were partially correct;

• 12 were partially incorrect; and

• 5 were incorrect

Additionally, one post was highlighted as ‘Neutral’, while ten were ‘Currently unverifiable’. The latter category pertains to posts written in the NOV-DEC period; too little time has passed to assess them.

Regardless, we were correct in 80% of claims and incorrect in 11%. Many claims were borderline, but we decided to mark them as partially incorrect rather than correct. Ultimately, the reader can decide for himself.

Given how little good quality information there is available on the war in Ukraine, internally, our ambition has been to achieve a score over 70%. As such, a score over 80% makes us very pleased. And I think it also highlights our continuous attempts to provide you with the best service.

To access the file with your assessments, please click here.

We anticipate that 2024 will continue to present challenges in the open-source landscape. While a substantial amount of data is available, there is also a constant effort on both sides to obscure the truth, disseminate propaganda, and spread disinformation.

If you would like to support us, please do so by clicking on the button below.

Konrad Muzyka – Rochan Consulting Director