Topics

At Liberogic, we are working to make our website structure more readable for AI agents and LLMs.

It has become commonplace for not only search engines, but also LLMs and AI agents like Claude, Gemini, and ChatGPT to read websites, organize and compare information, and conduct research on behalf of users.

On the Liberogic website, we're also preparing for AI to read our content by creating robots.txt, sitemap, link headers, llms.txt, and generating Markdown files for each article.

This time, Cloudflare released something two weeks ago.「Is Your Site Agent-Ready?」I will write an article about this matter.

Checked as a Content Site

The Liberogic website is a corporate and content site featuring company information, service introductions, news, columns, case studies, and more. It is neither an e-commerce site nor an API application requiring OAuth authentication.

The site type during server check is「Content Site」Select one.

Commerce, API/Auth/MCP-related items will be excluded from this evaluation, and we will primarily focus on the following aspects.

• Discoverability
• Content Accessibility
• Bot Access Control

As a result... the score is83点、Level 2「Bot-Aware」This is what happened 😭

Why isn't it a perfect score?!

Let's verify/confirm it right away.

In this check, both Discovery and Bot Access Control scored 100 points.

We have implemented support for items such as robots.txt, sitemap, link headers, AI bot rules, and Content Signals, and we have established the basic pathways for the AI ​​bot to discover the site and understand access policies.

To put it simply,

• Get people to discover the site
• Tell us where the content is located.
• This outlines the access policy for bots.
• Signals for AI

Those are the areas that have already been addressed.

On the other hand, I received a score of 0 for Content Accessibility (TT)

Looking at this alone, you might think, "Wait, I can't read the content?" but the reality is a little different.

The Markdown content itself is already prepared.

On the Liberogic website, a certain percentage of article content and case studies are already generated in Markdown format, which is easy for AI to read.
Both articles and case studies have Markdown files, and each page's head section contains a link tag to the Markdown content, providing a clear path for the AI ​​agent to access the Markdown files.

It's not that it doesn't support Markdown; rather, LLM is designed to retrieve the text in a readable format, so it's not a case of "not being ready to be read by AI."

So why don't I get a perfect score? 👺

The reason it didn't score 100 this time isn't because the site did nothing, but because of the incompatibility between the current implementation configuration and the evaluation specifications of the checking tool.

The Liberogic website is implemented using Astro's SSG, and the preview page is a hybrid design utilizing SSR.

While Markdown negotiation is typically handled by middleware, the hybrid configuration utilizes a Cloudflare adapter, and its output, _worker.js, takes precedence. As a result, the middleware fails to load, and the Markdown negotiation is not reflected in the checking tool.

In reality, there is a Markdown file, and it can be accessed via a link tag in the head section, but the checking tool was detecting it as "Markdown negotiation not being detected!"

Damn it! They made something half-baked!

...Although I'm tempted to say that, of course I rely on Cloudflare every day.

Thank you for approving our partner certification.

But as someone who has been striving for a perfect score in Lighthouse, this is a bit disappointing.

It's possible to get a perfect score! But I don't want to sacrifice operational efficiency or incur additional costs just for the sake of a high score.

There are several possible solutions.

• A separate Cloudflare Workers instance is set up to return Markdown.
• Convert the preview page to a CSR and move it closer to a complete SSG configuration.
• Change Cloudflare's service plan to a business plan and rewrite URLs using Transform Rules.

These are some of the methods.

That's right, upgrading Cloudflare's service level by one step would bring the score on the checking tool closer to 100.
However, at this point, deciding to pay extra just for that seems a bit questionable!

Our goal is not to get a perfect score on a checking tool, but rather to ensure that AI agents and LLMs can properly interpret the information on the Liberogic website. From the perspective of LLMs reading content, we believe we have made considerable progress in addressing the necessary steps.

Rather than increasing the cost just to get a perfect score, focus on being able to read properly.

It must be able to be operated properly.

As a Liberology company, we want to prioritize that balance.

Don't forget llms.txt!

Although it is not included in this checklist, at Liberology/llms.txtIt also supports this.

llms.txt is a file that acts like a table of contents for LLM, informing it "what information is on this site and which pages to read."

This concept is similar to sitemap.xml for search engines, but it's more suited to LLMs, serving to organize the site's overview and provide pathways to important content.

Although it's not included in the check tool's inspection items, Cloudflare prioritizes data quality, specifically "how well the content (body text) can be retrieved with minimal noise (i.e., whether it's in Markdown format)," so they may have intentionally omitted it.

Should I just let it read the HTML directly?

When having an AI agent read a website, you can, of course, have it analyze the HTML directly.

However, actual web pages contain a lot of information other than the main content, such as navigation, headers, footers, decorative elements, and JavaScript controls. Even pages that look natural often contain a lot of noise for LLM (Language-Likely Modulation).
Forcibly converting complex HTML to Markdown can disrupt heading structure, lists, and intended meaning, potentially resulting in data that is difficult for even LLMs to read.

While I'm saying this, I'm sure these issues will soon be improved with the advancement of AI, but I think it's important to do what's needed now, one step at a time. How far along are your websites in terms of AI implementation?