> For the complete documentation index, see [llms.txt](https://clients.medianova.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://clients.medianova.com/products/performance-cdn/static-content-delivery/advanced-configuration/caching/robots.txt-file.md).
# Robots.txt File
Define how the CDN serves the robots.txt file to control how search engines crawl and index your content.
**Robots.txt File** determines whether the CDN serves a `robots.txt` file and whether that file is provided by Medianova or retrieved from your origin server.
You can manage Robots.txt File settings in the [Medianova Control Panel](https://cloud.medianova.com) or via the [API](https://clients.medianova.com/api-documentation/performance-cdn/caching#put-api-v1-cdn-organization_uuid-resource-resource_uuid-7).
{% hint style="warning" %}
The selected mode can affect how search engines crawl and index your content. Verify the resulting `robots.txt` behavior before applying the configuration to production traffic.
{% endhint %}
## Configure Robots.txt File
{% stepper %}
{% step %}
### Open Robots.txt File
Open your CDN Resource and select **Caching**.
Locate **Robots.txt File**.
{% endstep %}
{% step %}
### Select the Mode
Select one of the following options:
* **Disabled** — No `robots.txt` file is served.
* **Enabled** — Medianova CDN serves its default `robots.txt` file. Custom `robots.txt` content cannot be uploaded through the Medianova Control Panel.
* **Origin** — The CDN retrieves and serves the `robots.txt` file from your origin server.
{% endstep %}
{% step %}
### Save the Configuration
Select **Save** in the pending changes bar.
Review the pending changes and confirm the update to apply the configuration to the CDN Resource.
{% endstep %}
{% endstepper %}
## Robots.txt Behavior
The selected mode determines how requests for `robots.txt` are handled.
When **Enabled** is selected, the CDN provides the Medianova default `robots.txt` file. To use your own crawler directives, select **Origin** and provide the `robots.txt` file from your origin server.
When **Origin** is selected and the origin does not provide a `robots.txt` file, the request returns `404 Not Found`.
{% hint style="info" %}
Robots.txt File controls the `robots.txt` response provided to web crawlers. It does not configure CDN caching or content delivery rules.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://clients.medianova.com/products/performance-cdn/static-content-delivery/advanced-configuration/caching/robots.txt-file.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
