Robots.txt

Muchinyorwa chino, tichaongorora basa rakakosha remarobots.txt faira mukugadzirisa traffic pamawebhusaiti, kurukura kukosha kwekuvapo kwayo, uye nekupa mazano ekuimisa kuti iite inoshanda peji indexing management. Pamusoro pezvo, tichaongorora mienzaniso yekushandiswa kwemirairi chaiyo murobhoti.txt faira uye topa gwara rekutarisa kurongeka kwezvirongwa zvayo.

Sei Robots.txt ichidiwa

Robots.txt ifaira riri pane saiti server mumudziyo wedhairekitori. Inozivisa injini yekutsvaga marobhoti maitiro avanofanirwa kuongorora zviri mukati meiyo sosi. Kushandiswa kwakakodzera kwefaira iyi kunobatsira kudzivirira kunyoreswa kwemapeji asingadiwe, inodzivirira zvakavanzika data, uye inogona kuvandudza kugona kweSEO optimization uye kuoneka kwesaiti mumigumo yekutsvaga. Kugadziriswa kwe robots.txt kunoitwa kuburikidza nemirairo, iyo yatichazotarisa mberi.

Kuisa Madhairekitori muRobots.txt

Mushandisi-Mumiriri

Iyo yekutanga dhairekitori inozivikanwa seMushandisi-Mumiriri, kwatinoisa yakakosha kiyi izwi remarobhoti. Pakuona izwi iri, iro robhoti rinonzwisisa kuti mutemo unoitirwa iwo chaiwo.

Funga muenzaniso wekushandisa User-Agent murobhoti.txt faira:

User-Agent: *
Disallow: /private/

Muenzaniso uyu unoratidza kuti marobhoti ese ekutsvaga (anomiririrwa nechiratidzo " * ") anofanira kufuratira mapeji ari mudhairekitori re /private/.

Heano maitiro ekurairirwa anotarisa kune chaiwo ekutsvaga marobhoti:

User-Agent: Googlebot
Disallow: /admin/

User-Agent: Bingbot
Disallow: /private/

Muchiitiko ichi, robhoti yekutsvaga yeGooglebot inofanira kufuratira mapeji ari mudhairekitori re /admin/ , ukuwo Bingbot ichifuratira mapeji ari mudhairekitori re /private/.

Usabvumira

Usabvumira unoudza marobhoti ekutsvaga kuti ndeapi maURL ekusaisa kana kusaisa mu index pawebhusaiti. Murairo uyu unobatsira kana uchida kuvanza data rakavanzika kana mapeji ezvinyorwa zvemhando yepasi kuti asaiswa mu index nemainjini ekutsvaga. Kana faira re robots.txt riine chinyorwa chinonzi Usabvumira: /directory/ , saka marobhoti acharambidzwa kupinda muzviri mu directory rataurwa. Semuenzaniso,

User-agent: *
Disallow: /admin/

Kukosha uku kunoratidza kuti marobhoti ese anofanira kufuratira maURL anotanga na /admin/ . Kuti udzivise webhusaiti yese kuti isaverengerwe nemarobhoti, isa dhairekitori remidzi semutemo:

User-agent: *
Disallow: /

tendera

"Bvumira" kukosha kunoita zvinopesana ne "Rega kubvumidza": inobvumira kutsvaga marobhoti kuwana kune rimwe peji kana dhairekitori, kunyangwe zvimwe mirairo murobhoti.txt faira inorambidza kupinda kwairi.

Chimbofunga muenzaniso:

User-agent: *
Disallow: /admin/
Allow: /admin/login.html

Mumuenzaniso uyu, zvinotsanangurwa kuti marobhoti haabvumidzwe kupinda mudhairekitori re /admin/ , kunze kwepeji re /admin/login.html , iro rinowanikwa kuti riiswe mu indexing uye rionekwe.

Robots.txt uye Sitemap

Sitemap ifaira reXML rine runyoro rwema URL emapeji ese uye mafaera ari pasaiti anogona kunyoreswa neinjini dzekutsvaga. Kana robhoti rekutsvaga richinge rawana marobhoti.txt faira roona chinongedzo kune saitimepu XML faira, rinogona kushandisa faira iri kuwana ese aripo maURL nezviwanikwa pasaiti. Murairo unotsanangurwa muchimiro:

Sitemap: https://yoursite.com/filesitemap.xml

Uyu mutemo unowanzoiswa pamagumo egwaro pasina kusungirirwa kune chaiyo Mushandisi-Agent uye inogadziriswa nemarobhoti ese pasina kusarudzika. Kana muridzi wesaiti akasashandisa sitemap.xml, hazvifanirwe kuwedzera mutemo.

Mienzaniso yeConfigured Robots.txt

Kumisikidza Robots.txt yeWordPress

Muchikamu chino, isu tichatarisa yakagadzirira-yakagadzirwa gadziriso yeWordPress. Isu tichaongorora kuvharidzira kupinda kune zvakavanzika data uye kubvumidza kupinda kune iwo makuru mapeji.

Semhinduro yakagadzirira, unogona kushandisa kodhi inotevera:

User-agent: *
# Block access to files containing confidential data
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php

# Allow access to the main site pages
Allow: /wp-content/uploads/
Allow: /sitemap.xml
Allow: /feed/
Allow: /trackback/
Allow: /comments/feed/
Allow: /category/*/*
Allow: /tag/*

# Prohibit the indexing of old versions of posts and parameterized queries to avoid content duplication or suboptimal indexing.
Disallow: /*?*
Disallow: /?s=*
Disallow: /?p=*
Disallow: /?page_id=*
Disallow: /?cat=*
Disallow: /?tag=*

# Include the sitemap (location needs to be replaced with your own)
Sitemap: http://yourdomain.com/sitemap.xml

Kunyangwe zvese zvinongedzo zvichiperekedzwa nemhinduro, ngationgororei zvakadzama mumhedziso.

1. Marobhoti haaratidze mafaera anonzwisisika uye madhairekitori.
2. Panguva imwecheteyo, marobhoti anobvumirwa kupinda kune mapeji makuru uye zviwanikwa zvesaiti.
3. kurambidzwa kunoiswa pakunongedza zvinyorwa zvekare zvezvinyorwa uye mibvunzo ine parameterized kudzivirira kudzokororwa kwemukati.
4. Nzvimbo yesaitimepu inoratidzirwa kune yakavandudzwa indexing.

Nokudaro, takatarisa muenzaniso wakakwana wegadziriro yakagadzirira, umo mamwe mafaira akajeka uye nzira dzakavanzwa kubva kune indexing, asi magwaro makuru anowanikwa.

Kusiyana nemawebhusaiti mazhinji anozivikanwa eCMS kana akanyorwa nemaoko, WordPress ine maplugins akati wandei anobatsira kugadzira nekutarisira faira re robots.txt. Imwe yemhinduro dzakakurumbira dzechinangwa ichi iYoast SEO.

Kuti uiise, unoda:

1. Enda kune WordPress admin panel.
2. Muchikamu che "Plugins", sarudza "Wedzera Nyowani".
3. Tsvaga iyo "Yoast SEO" plugin uye uiise.
4. Shandisai plugin.

Kuti ugadzirise iro robots.txt faira, unofanira:

1. Enda kuchikamu che "SEO" mune iyo admin panel's side menu uye sarudza "General".
2. Enda kune "Tools" tab.
3. Dzvanya pa "Files". Pano uchaona mafaira akasiyana-siyana, kusanganisira robots.txt.
4. Isa mitemo inodiwa ye indexing zvinoenderana nezvaunoda.
5. Mushure mekuita shanduko kufaira, tinya bhatani "Chengetedza shanduko ku robots.txt".

Ziva kuti imwe neimwe robots.txt faira yekumisikidza yeWordPress yakasarudzika uye zvinoenderana nezvinodiwa uye maficha esaiti. Iko hakuna universal template ingakodzera zvese zviwanikwa pasina kusarudzika. Nekudaro, uyu muenzaniso uye kushandiswa kwema plugins kunogona kurerutsa basa.

Manual Setting yeRobots.txt

Saizvozvo, iwe unogona kuseta yako kumisikidzwa kwefaira kunyangwe pasina CMS yakagadzirira yesaiti. Mushandisi anodawo kuisa iyo robots.txt faira kumudzi wedhairekitori resaiti uye tsanangura mitemo inodiwa. Heino mumwe wemienzaniso, umo mirairo yese iripo inoratidzwa:

User-agent: *
Disallow: /admin/             # Prohibit access to the administrative panel
Disallow: /secret.html	      # Prohibit access to a specific file
Disallow: /*.pdf$	      # Prohibit indexing of certain file types
Disallow: /*?sort=	      # Prohibit indexing of certain URL parameters
Allow: /public/		      # Allow access to public pages
Sitemap: http://yourdomain.com/sitemap.xml # Include the sitemap

Maonero eRobots.txt File

Sechishandiso chekubatsira paunotarisa robots.txt faira yezvikanganiso, zvinokurudzirwa kushandisa mabasa epaIndaneti.

Funga nezvemuenzaniso weYandex Webmaster service. Kuti utarise, unofanirwa kuisa link kune yako saiti mumunda unoenderana kana faira racho ratoiswa kuseva. Mushure meizvozvo, chishandiso pachacho chichaisa configuration yefaira. Panewo sarudzo yekuisa configuration nemaoko:

Tevere, iwe unofanirwa kukumbira cheki uye kumirira mhinduro:

Mumuenzaniso wakapihwa, hapana zvikanganiso. Kana paine, sevhisi icharatidza nzvimbo dzine dambudziko uye nzira dzekudzigadzirisa.

mhedziso

Muchidimbu, takasimbisa kukosha kweiyo robots.txt faira pakudzora traffic panzvimbo. Isu takapa zano rekuti tingaimisa sei nemazvo kubata sei injini dzekutsvaga mapeji. Pamusoro peizvi, takatarisawo mienzaniso yemashandisiro ekushandisa faira iri nemazvo uye tikapa mirairo yekutarisa kuti ese magadzirirwo ari kushanda nemazvo.