Definition
robots.txt: A text file at the root of a site that tells crawlers which URLs they may crawl. It controls crawling, not indexing, and is not a security tool.
Definition from the OneCity AI Academy course notes.
Where you learn it
- Module 3: Search Engine Optimisation (SEO)
- Class: How Google Works: Crawling, Indexing & Ranking
Related terms
Learn robots.txt properly, with your own AI tutor
Module 1 is a free trial. Any paid plan includes all 16 modules, the AI tutor, quizzes and graded practice tasks.
- Monthly₹1,0001 month access
- 3 Months · best value₹2,5003 months access
One-time payments · no auto-debit · WhatsApp us