The purpose of this experiment was to use the technology to address reader pains in the existing news experience, and in particular, to start Ekstra Bladet’s journey towards becoming AI-driven.

The project, titled Platform Intelligence in News (PIN), aimed to leverage AI to reach three strategic goals, said Kasper Lindskow, Head of AI, JP/Politikens Media Group. 

• To increase editorial and commercial value creation by using AI to create a new product iteration of ekstrabladet.dk with a wider, deeper, and richer news experience.
• Implement AI systems that align with editorial values, and maintain control over these technologies.
• Promote healthy norms for AI use in media by sharing their experiences and insights from achieving the first two goals.

Ekstra Bladet was our winner for Best Use of AI in Revenue Strategy at WAN-IFRA’s Digital Media Awards Worldwide. Download our Best in Innovation Report, free for Members.

“The aim was to use AI to strengthen editorial mission and business strategy. Our editorial mission would be strengthened by creating a wider, deeper and richer news experience where readers would get more relevant news in line with our editorial profile,” Lindskow said. 

“And, our business strategy would be supported via increasing advertising revenues and a growing subscriber base resulting from increased traffic and subscriptions sales and reduced churn,” he added.

Editor’s note: In late 2024, J/P Politikens published a freely available report detailing key learnings during the first 4-5 years of PIN. You can access it here.

AI enables a wholesome news experience

For this project, the team focussed on three types of AI to create a new news experience.

1. Wider News Experience: Recommender Systems

Ekstra Bladet implemented AI-driven recommender systems to help readers discover relevant news in a fast-paced digital environment. Unlike entertainment platforms, where content remains relevant longer, news articles quickly lose relevance. 

The PIN team designed recommendation models that balance popularity and personalisation while ensuring editorial integrity and avoiding filter bubbles.

Technical implementation

Ekstra Bladet’s recommender system architecture comprises multiple models:

• Collaborative filtering models (x4): These models predict user preferences based on shared behaviors across similar users. While effective at surfacing popular content, they introduced a bias toward trending stories.
• Content-based filtering models (x2): These models analysed article metadata, using topic classification, Named Entity Recognition (NER), and sentiment analysis to recommend articles based on individual reading history. This approach helped surface niche content but required mitigation strategies to prevent excessive reinforcement of user preferences.
• Hybrid models (x2): By combining collaborative and content-based filtering, these models aimed to strike a balance between discoverability and personalisation.
• Contrastive learning-based similarity models: These models assessed article similarity by leveraging deep learning techniques, ensuring recommendations reflected thematic coherence rather than simple keyword matching.

Example of the different news flow effects of two well-optimised recommender systems. Source:. A/B test of recommendation methods on ekstrabladet.dk from the PIN-project.

Ekstra Bladet implemented six horizontal swipeable sections on its homepage, each driven by different AI models:

• Top sections: Collaborative filtering models that surface popular, recently published news stories based on user interactions.
• Middle sections: Content-based filtering models tailored to reader histories, ensuring niche story exposure.
• Bottom sections: Hybrid models that balance personalisation and diversity.

Results and insights

The implementation of these systems yielded substantial engagement improvements:

• 110 percent increase in free article consumption
• 38 percent increase in paid article readership
• 35 percent increase in subscription conversions

A/B testing provided key insights into the performance of different models. Collaborative filtering models generated high engagement but favored already popular stories. 

Different recommender systems suggest personalised articles to various reader segments on the home page.

Content-based models improved niche discovery but risked creating filter bubbles. Hybrid models offered the most balanced approach, mitigating bias while enhancing personalisation.

2. Deeper News Experience: Natural Language Processing (NLP)

To deepen the reading experience, Ekstra Bladet leveraged NLP to enhance content discovery, automate metadata tagging, and improve article recommendations. The primary goal was to surface related news stories dynamically, enabling readers to explore topics with greater depth without relying on explicit user preferences.

Technical implementation

The PIN team deployed various NLP models, each fine-tuned for specific news-related tasks:

• Topic classification models: These transformer-based neural networks categorised articles according to taxonomies like IAB and IPTC, enabling structured classification of news topics.
• Named entity recognition (NER): The system automatically identified and tagged entities (e.g., people, organizations, locations) within articles. This facilitated better indexing and retrieval of related content.
• Sentiment analysis models: These models assessed the emotional tone of articles, aiding in content curation and semantic ad targeting.
• Contrastive learning-based similarity models: These deep-learning models significantly improved related article recommendations by evaluating contextual similarity beyond mere keyword overlap.

Related articles on ekstrabladet.dk’s article page identified with text similarity models based on contrastive learning.

Results and insights

Ekstra Bladet’s NLP-driven improvements led to a 120 percent increase in traffic from related article suggestions. Key achievements included:

• A shift from word2vec to DaBERT-Contrastive models, increasing the click-through rate (CTR) from related article recommendations from 7.15 percent to 8.61 percent.

• More accurate entity recognition, enhancing topic-based personalisation and allowing readers to follow topics of interest dynamically.
• Improved semantic ad targeting, enabling privacy-friendly ad placement without user tracking.

The NLP framework also automated content tagging and discovery, making older articles more accessible and increasing the longevity of newsroom content. Furthermore, semantic search capabilities allowed users to follow topics of interest dynamically, boosting engagement.

3. Richer News Experience: Natural Language Generation (NLG)

Ekstra Bladet’s use of NLG focused on expanding local news coverage through automation. The goal was to enhance the volume and depth of service journalism while maintaining editorial oversight.

Technical implementation

The team developed MAGNA (Monitoring and Assisted Generation of News Artefacts), a multi-phase AI-driven editorial toolset. The system evolved through several iterations:

• Rule-based automation (GOFAI): Initially, Ekstra Bladet used AX Semantics’ rule-based system to generate short local news reports on train delays, weather, corporate financials, and real estate transactions. By 2021, 200,000 AI-generated articles had been published.
• Hybrid AI summarisation (2023): OpenAI’s GPT models were integrated to generate summaries and improve article coherence. Retrieval-Augmented Generation (RAG) ensured AI-generated content was grounded in factual data from Ekstra Bladet’s archives, reducing hallucinations.
• Human-in-the-loop review: Editors reviewed AI-generated summaries before publication. A feedback mechanism allowed rejected summaries to inform future iterations, improving accuracy.

Ekstra Bladet also introduced ZIP code-based local news, where readers could enter their ZIP code to access an AI-curated mix of AI-generated and human-written stories.

Results and insights

• 289 AI-enhanced articles were published in 2023
• The AI-assisted headline generation tool improved engagement by dynamically refining article titles based on content summaries
• Human-in-the-loop verification significantly improved the factual reliability of AI-generated content, reducing errors from 41 percent to under 10 percent after integrating rejection-based retraining.
• Journalists saved time and focused on in-depth reporting, as AI-assisted tools handled routine content generation.

Ekstra Bladet’s newsroom began testing MAGNA in October 2023, and its adoption increased as journalists found value in its features, particularly in headline generation and summarisation tools. Journalists particularly embraced AI-assisted headline generation and automated fact boxes, viewing these tools as collaborative assets rather than replacements.

Building comprehensive AI systems in-house

To ensure value alignment, a core goal of PIN, the team developed its core machine learning technology in-house. They also established partnerships with universities, giving access to the competencies needed to develop and evaluate their AI systems holistically. Here’s how they went about it:

• While configuring the mix of recommender systems for large scale implementation in 2023, the team relied on insights from 50+ A/B tests of the effects of different systems on the news flow. 

“These tests had, among other things, revealed that our collaborative filtering systems had a significant bias towards the most popular news stories, while our content-based recommenders had a low ability to surprise readers with new stories outside their usual reading diet – qualities that we sought to balance when implementing recommender systems at scale,” Lindskow said.

• When training and deploying NLP-models, they relied on knowledge from an evaluation framework developed on PIN that identified the effects of different training methods on the accuracy, efficiency and utility bias of the models. 

“This allowed us to deploy NLP-models with a balanced tradeoff between accuracy and efficiency (costs and CO2 emissions) that were not biased against any specific group,” he said.

• Finally, when developing AI systems based on natural language generation, they employed a mix of manual and automated tests to minimise factual errors and hallucinations from entering the news flow. 

Lindskow said insights from these tests prompted them to introduce a human-in-the-loop for all AI-generated explanatory boxes and to ensure that all generative editorial tools are anchored in external facts, like the article archive, rather than relying on the internal “knowledge” of GPT-4.

This story is also featured in our Digital Revenue newsletter. Find out more about our newsletters and sign up for them here.

To achieve the goal of healthy norm-setting for AI in news, Ekstra Bladet openly shares its learnings from PIN.

“We open sourced some of our NLP-algorithms, and in March this year, open sourced our basic recommendation algorithms, along with a training set for the development of responsible news recommenders,” he noted. 

Editorial values and reader needs: Lessons learned

PIN has involved navigating in uncharted territory with lots of A/B testing. Lindskow shared a few of the team’s learnings: 

• Testing methods from other media sectors often proves suboptimal for news publishing. For example, Netflix’s collaborative filtering struggles with “cold start” issues in fast-paced news environments. Instead, deep content-based recommender systems using detailed NLP signals from news articles generally perform better for news recommendations.
• AI systems are not neutral and their design choices greatly influence outcomes. For instance, while recommender systems may equally drive readership, they can impact news flow and reader demographics differently. They, therefore, use a mix of systems to align with their editorial values and address diverse reader needs.
• In late summer 2023, they discovered that generative AI, like ChatGPT, rarely hallucinates when anchored in external knowledge bases. Consequently, they ground their generative editorial tools in external facts.