Explainers
What is Robots.txt?
Robots.txt guides crawler access per path — advisory file, not security.
<
p
c
l
a
s
s
=
"
m
t
-
4
l
e
a
d
i
n
g
-
7
t
e
x
t
-
s
l
a
t
e
-
7
0
0
"
>
U
s
e
i
t
t
o
c
a
l
m
c
r
a
w
l
w
a
s
t
e
(
a
d
m
i
n
,
f
i
l
t
e
r
s
)
;
n
e
v
e
r
t
o
h
i
d
e
p
r
i
v
a
t
e
d
a
t
a
.
P
e
o
p
l
e
i
g
n
o
r
e
r
o
b
o
t
s
;
a
t
t
a
c
k
e
r
s
l
o
v
e
f
i
n
d
i
n
g
t
h
e
m
.
<
/
p
>
<
h
2
c
l
a
s
s
=
"
m
t
-
1
0
t
e
x
t
-
2
x
l
f
o
n
t
-
b
o
l
d
t
e
x
t
-
s
l
a
t
e
-
9
5
0
"
>
S
e
e
a
l
s
o
<
/
h
2
>
<
u
l
c
l
a
s
s
=
"
m
t
-
4
l
i
s
t
-
d
i
s
c
p
l
-
6
l
e
a
d
i
n
g
-
7
t
e
x
t
-
s
l
a
t
e
-
7
0
0
"
>
<
l
i
c
l
a
s
s
=
"
m
t
-
2
"
>
<
a
c
l
a
s
s
=
"
t
e
x
t
-
i
n
d
i
g
o
-
7
0
0
u
n
d
e
r
l
i
n
e
"
h
r
e
f
=
"
/
w
e
b
s
i
t
e
-
l
a
u
n
c
h
-
c
h
e
c
k
l
i
s
t
-
i
n
d
i
a
/
"
>
L
a
u
n
c
h
c
h
e
c
k
l
i
s
t
<
/
a
>
<
/
l
i
>
<
l
i
c
l
a
s
s
=
"
m
t
-
2
"
>
<
a
c
l
a
s
s
=
"
t
e
x
t
-
i
n
d
i
g
o
-
7
0
0
u
n
d
e
r
l
i
n
e
"
h
r
e
f
=
"
/
g
l
o
s
s
a
r
y
/
s
e
o
-
t
e
r
m
s
-
g
l
o
s
s
a
r
y
.
h
t
m
l
"
>
S
E
O
g
l
o
s
s
a
r
y
<
/
a
>
<
/
l
i
>
<
/
u
l
>
FAQs
Do I need to understand Robots.txt?
Enough to brief developers and read proposals — this page plus our SEO glossary covers it.
Where to learn more?
Related links above, or ask CBWS on WhatsApp.
Ready to Transform Your Business?
Let's discuss your project requirements and create something amazing together.