Ma kēia ʻatikala, e nānā mākou i ke kuleana koʻikoʻi o ka faila robots.txt i ka mālama ʻana i nā kaʻa ma nā pūnaewele, kūkākūkā i ka pono o kona hele ʻana, a hāʻawi i nā ʻōlelo aʻoaʻo no ka hoʻonohonoho ʻana no ka hoʻokele kuhikuhi ʻaoʻao kūpono. Eia hou, e kālailai mākou i nā hiʻohiʻona o ka hoʻohana ʻana i nā kuhikuhi pololei ma ka faila robots.txt a hāʻawi i kahi alakaʻi i ka nānā ʻana i ka pololei o kāna mau hoʻonohonoho.
No ke aha e pono ai ʻo Robots.txt
ʻO Robots.txt kahi faila i loaʻa ma ka kikowaena pūnaewele ma kāna papa kuhikuhi kumu. Hoʻomaopopo ia i nā lopako ʻenekini hulina pehea e nānā pono ai lākou i ka ʻike o ka punawai. ʻO ka hoʻohana pono ʻana i kēia faila e kōkua i ka pale ʻana i ka helu ʻana i nā ʻaoʻao makemake ʻole, pale i ka ʻikepili huna, a hiki ke hoʻomaikaʻi i ka pono o ka SEO optimization a me ka ʻike ʻana o ka pūnaewele i nā hopena hulina. Hana ʻia ka hoʻonohonoho o robots.txt ma o nā kuhikuhi, a mākou e nānā hou aku ai.
Hoʻonohonoho i nā kuhikuhi ma Robots.txt
Mea Hoʻohui Mea Pili
Ua ʻike ʻia ke kuhikuhi mua ʻo User-Agent, kahi i hoʻonohonoho ai mākou i kahi huaʻōlelo kūikawā no nā robots. I ka ʻike ʻana i kēia huaʻōlelo, hoʻomaopopo ka robot ua manaʻo ʻia ka lula no ia.
E noʻonoʻo i kahi laʻana o ka hoʻohana ʻana i ka User-Agent ma ka faila robots.txt:
User-Agent: *
Disallow: /private/Hōʻike kēia hiʻohiʻona i nā robots hulina āpau (hōʻike ʻia e ka hōʻailona "*") pono e haʻalele i nā ʻaoʻao i loaʻa i ka / pilikino/ papa kuhikuhi.
Eia ke ʻano o ke aʻo ʻana i nā lopako hulina kikoʻī.
User-Agent: Googlebot
Disallow: /admin/
User-Agent: Bingbot
Disallow: /private/I kēia hihia,ʻo Googlebot ʻO ka lopako huli pono e haʻalele i nā ʻaoʻao ma ka /admin/ papa kuhikuhi, oiai Bingbot pono e haʻalele i nā ʻaoʻao ma ka / pilikino/ papa kuhikuhi.
hōʻole aku
hōʻole aku haʻi i nā lopako huli i nā URL e lele ai a ʻaʻole kuhikuhi ma ka pūnaewele. Pono kēia kuhikuhi inā makemake ʻoe e hūnā i nā ʻikepili koʻikoʻi a i ʻole nā palapala maʻiʻo haʻahaʻa mai ka helu ʻia ʻana e nā ʻenekini huli. Inā loaʻa i ka faila robots.txt ke komo ʻAʻole ʻae: /directory/, a laila e hōʻole ʻia nā robots i ke komo ʻana i nā mea o ka papa kuhikuhi i kuhikuhi ʻia. ʻo kahi laʻana,
User-agent: *
Disallow: /admin/Hōʻike kēia waiwai i kēlā nā lopako a pau pono e haʻalele i nā URL e hoʻomaka ana me /admin/. No ka pale ʻana i ka pūnaewele holoʻokoʻa mai ka helu ʻia ʻana e nā robots, e hoʻonohonoho i ka papa kuhikuhi kumu ma ke ʻano he lula:
User-agent: *
Disallow: /ae aku
ʻO ka waiwai "Allow" e kū'ē ana i ka "Disallow": hiki i nā lopako huli ke komo i kahi ʻaoʻao a i ʻole papa kuhikuhi kikoʻī, ʻoiai inā pāpā ʻia nā kuhikuhi ʻē aʻe i ka faila robots.txt ke komo iā ia.
E noʻonoʻo i kekahi laʻana:
User-agent: *
Disallow: /admin/
Allow: /admin/login.htmlMa kēia laʻana, ua ʻōlelo ʻia ʻaʻole ʻae ʻia nā robots e komo i ka /admin/ papa kuhikuhi, koe wale no ka /admin/login.html ʻaoʻao, i loaʻa no ka helu ʻana a me ka nānā ʻana.
Robots.txt a me Sitemap
He waihona XML ka Sitemap kahi papa inoa o nā URL o nā ʻaoʻao a me nā faila ma ka pūnaewele i hiki ke kuhikuhi ʻia e nā ʻenekini huli. Ke komo ka lopako huli i ka faila robots.txt a ʻike i kahi loulou i kahi faila XML sitemap, hiki iā ia ke hoʻohana i kēia faila e ʻimi i nā URL a me nā kumuwaiwai āpau i loaʻa ma ka pūnaewele. Ua kuhikuhi ʻia ke kuhikuhi ma ke ʻano:
Sitemap: https://yoursite.com/filesitemap.xmlHoʻokomo pinepine ʻia kēia lula ma ka hope o ka palapala me ka nakinaki ʻole ʻia i kahi mea hoʻohana-Agent kikoʻī a ua hana ʻia e nā robots āpau me ka ʻole. Inā ʻaʻole hoʻohana ka mea nona ka pūnaewele sitemap.xml, ʻaʻole pono e hoʻohui i ka lula.
Nā laʻana o Robots.txt i hoʻonohonoho ʻia
Hoʻonohonoho i Robots.txt no WordPress
Ma kēia ʻāpana, e noʻonoʻo mākou i kahi hoʻonohonoho mākaukau no WordPress. E ʻimi mākou i ka pale ʻana i ka ʻike huna a me ka ʻae ʻana i ke komo ʻana i nā ʻaoʻao nui.
Ma ke ʻano he hopena mākaukau, hiki iā ʻoe ke hoʻohana i kēia code:
User-agent: *
# Block access to files containing confidential data
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
# Allow access to the main site pages
Allow: /wp-content/uploads/
Allow: /sitemap.xml
Allow: /feed/
Allow: /trackback/
Allow: /comments/feed/
Allow: /category/*/*
Allow: /tag/*
# Prohibit the indexing of old versions of posts and parameterized queries to avoid content duplication or suboptimal indexing.
Disallow: /*?*
Disallow: /?s=*
Disallow: /?p=*
Disallow: /?page_id=*
Disallow: /?cat=*
Disallow: /?tag=*
# Include the sitemap (location needs to be replaced with your own)
Sitemap: http://yourdomain.com/sitemap.xmlʻOiai ua hui pū ʻia nā kuhikuhi a pau me nā manaʻo, e ʻimi hohonu kākou i nā hopena.
- ʻAʻole nā robots e kuhikuhi i nā faila a me nā papa kuhikuhi.
- I ka manawa like, ʻae ʻia nā robots e komo i nā ʻaoʻao nui a me nā kumuwaiwai o ka pūnaewele.
- Ua hoʻonoho ʻia ka pāpā i ka helu ʻana i nā mana kahiko o nā pou a me nā nīnau i hoʻohālikelike ʻia e pale i ka hoʻopili hou ʻana.
- Hōʻike ʻia ka wahi o ka palapala ʻāina no ka hoʻomaikaʻi ʻana i ka kuhikuhi ʻana.
No laila, ua noʻonoʻo mākou i kahi hiʻohiʻona maʻamau o kahi hoʻonohonoho mākaukau, kahi i hūnā ʻia ai kekahi mau faila a me nā ala mai ka indexing, akā hiki ke loaʻa nā papa kuhikuhi nui.
ʻAʻole like me nā CMS kaulana a i ʻole nā pūnaewele i kākau maʻamau, loaʻa i ka WordPress kekahi mau plugins e hoʻomaʻamaʻa i ka hana ʻana a me ka hoʻokele ʻana i ka faila robots.txt. ʻO kekahi o nā hoʻonā kaulana no kēia kumu Yoast SEO.
No ka hoʻouka ʻana, pono ʻoe:
- E hele i ka WordPress admin panel.
- Ma ka ʻāpana "Plugins", koho "Add New".
- E ʻimi i ka plugin "Yoast SEO" a hoʻokomo iā ia.
- E hoʻopili i ka plugin.
No ka hoʻoponopono ʻana i ka faila robots.txt, pono ʻoe e:
- E hele i ka ʻāpana "SEO" ma ka ʻaoʻao ʻaoʻao o ka admin panel a koho i ka "General".
- E hele i ka "Tools" tab.
- Kaomi ma ka "Files". Maanei ʻoe e ʻike ai i nā faila like ʻole, me robots.txt.
- E hoʻokomo i nā lula kuhikuhi pono e like me kāu mau koi.
- Ma hope o ka hoʻololi ʻana i ka faila, kaomi i ke pihi "Save changes to robots.txt".
E hoʻomaopopo he ʻokoʻa kēlā me kēia hoʻonohonoho faila robots.txt no WordPress a pili i nā pono kikoʻī a me nā hiʻohiʻona o ka pūnaewele. ʻAʻohe kumu hoʻohālike āpau e kūpono i nā kumuwaiwai āpau me ka ʻole. Eia naʻe, hiki i kēia hiʻohiʻona a me ka hoʻohana ʻana i nā plugins ke maʻalahi i ka hana.
Hoʻonohonoho lima o Robots.txt
Pēlā nō, hiki iā ʻoe ke hoʻonohonoho i kāu hoʻonohonoho o ka faila inā ʻaʻole i loaʻa kahi CMS mākaukau no ka pūnaewele. Pono ka mea hoʻohana e hoʻouka i ka faila robots.txt i ka papa kuhikuhi kumu o ka pūnaewele a kuhikuhi i nā lula e pono ai. Eia kekahi o nā laʻana, kahi i hōʻike ʻia ai nā kuhikuhi āpau i loaʻa:
User-agent: *
Disallow: /admin/ # Prohibit access to the administrative panel
Disallow: /secret.html # Prohibit access to a specific file
Disallow: /*.pdf$ # Prohibit indexing of certain file types
Disallow: /*?sort= # Prohibit indexing of certain URL parameters
Allow: /public/ # Allow access to public pages
Sitemap: http://yourdomain.com/sitemap.xml # Include the sitemapPehea e nānā ai i ka faila Robots.txt
Ma ke ʻano he mea kōkua i ka nānā ʻana i ka faile robots.txt no nā hewa, ʻōlelo ʻia e hoʻohana i nā lawelawe pūnaewele.
E noʻonoʻo i ka laʻana o ka Luna Pūnaewele ʻo Yandex lawelawe. No ka nānā ʻana, pono ʻoe e hoʻokomo i kahi loulou i kāu pūnaewele ma ke kahua kūpono inā ua hoʻoili ʻia ka faila i ke kikowaena. Ma hope o kēlā, e hoʻouka ka mea hana i ka hoʻonohonoho faila. Aia kekahi koho e hoʻokomo i ka hoʻonohonoho me ka lima:
A laila, pono ʻoe e noi i kahi loiloi a kali i nā hopena:
I ka laʻana i hāʻawi ʻia, ʻaʻohe hewa. Inā loaʻa kekahi, e hōʻike ka lawelawe i nā wahi pilikia a me nā ala e hoʻoponopono ai.
Panina
I ka hōʻuluʻulu manaʻo, ua hoʻoikaika mākou i ke koʻikoʻi o ka faila robots.txt no ka hoʻokele ʻana i ke kaʻa ma ka pūnaewele. Hāʻawi mākou i ka ʻōlelo aʻoaʻo e pili ana i ka hoʻonohonoho pono ʻana i ka hoʻokele ʻana i nā ʻaoʻao index engines hulina. Ma waho aʻe o kēia, ua nānā mākou i nā hiʻohiʻona o ka hoʻohana pono ʻana i kēia faila a hāʻawi i nā ʻōlelo aʻoaʻo i ka nānā ʻana i ka hana pololei ʻana o nā hoʻonohonoho āpau.
