Robots.txt

Muchinyorwa chino, tichaongorora basa rakakosha remarobots.txt faira mukugadzirisa traffic pamawebhusaiti, kurukura kukosha kwekuvapo kwayo, uye nekupa mazano ekuimisa kuti iite inoshanda peji indexing management. Pamusoro pezvo, tichaongorora mienzaniso yekushandiswa kwemirairi chaiyo murobhoti.txt faira uye topa gwara rekutarisa kurongeka kwezvirongwa zvayo.

Sei Robots.txt ichidiwa

Robots.txt ifaira riri pane saiti server mumudziyo wedhairekitori. Inozivisa injini yekutsvaga marobhoti maitiro avanofanirwa kuongorora zviri mukati meiyo sosi. Kushandiswa kwakakodzera kwefaira iyi kunobatsira kudzivirira kunyoreswa kwemapeji asingadiwe, inodzivirira zvakavanzika data, uye inogona kuvandudza kugona kweSEO optimization uye kuoneka kwesaiti mumigumo yekutsvaga. Kugadziriswa kwe robots.txt kunoitwa kuburikidza nemirairo, iyo yatichazotarisa mberi.

Kuisa Madhairekitori muRobots.txt

Mushandisi-Mumiriri

Iyo yekutanga dhairekitori inozivikanwa seMushandisi-Mumiriri, kwatinoisa yakakosha kiyi izwi remarobhoti. Pakuona izwi iri, iro robhoti rinonzwisisa kuti mutemo unoitirwa iwo chaiwo.

Funga muenzaniso wekushandisa User-Agent murobhoti.txt faira:

User-Agent: *
Disallow: /private/

Uyu muenzaniso unoratidza kuti ese ekutsvaga marobhoti (anomiririrwa nechiratidzo "*") inofanirwa kufuratira mapeji ari mu /yakavanzika/ dhairetori.

Heano maitiro ekurairirwa anotarisa kune chaiwo ekutsvaga marobhoti:

User-Agent: Googlebot
Disallow: /admin/

User-Agent: Bingbot
Disallow: /private/

Munyaya iyi, the googlebot tsvaga robhoti inofanira kufuratira mapeji mu /admin/ directory, apo bingbot inofanira kufuratira mapeji mu /yakavanzika/ dhairetori.

Usabvumira

Usabvumira inoudza marobhoti ekutsvaga ma URL ekusvetuka kana kusaisa index pawebhusaiti. Iyi dhairekitori inobatsira kana iwe uchida kuvanza data rakavanzika kana yakaderera-mhando mapeji emukati kuti asaiswa indexed neinjini dzekutsvaga. Kana iro robots.txt faira rine chinopinda Rega: /dhairekitori/, ipapo marobhoti acharambidzwa kuwana zviri mukati medhairekitori rakataurwa. Semuyenzaniso,

User-agent: *
Disallow: /admin/

Kukosha uku kunoratidza kuti marobhoti ose inofanira kufuratira maURL kutanga /admin/. Kuvharisa iyo saiti yese kubva pakunyorwa nechero marobhoti, isa iyo midzi dhairekitori semutemo:

User-agent: *
Disallow: /

tendera

"Bvumira" kukosha kunoita zvinopesana ne "Rega kubvumidza": inobvumira kutsvaga marobhoti kuwana kune rimwe peji kana dhairekitori, kunyangwe zvimwe mirairo murobhoti.txt faira inorambidza kupinda kwairi.

Chimbofunga muenzaniso:

User-agent: *
Disallow: /admin/
Allow: /admin/login.html

Mumuenzaniso uyu, zvinotsanangurwa kuti marobhoti haabvumirwe kupinda kune /admin/ directory, kunze kwe /admin/login.html peji, inowanika ku indexing uye kuongorora.

Robots.txt uye Sitemap

Sitemap ifaira reXML rine runyoro rwema URL emapeji ese uye mafaera ari pasaiti anogona kunyoreswa neinjini dzekutsvaga. Kana robhoti rekutsvaga richinge rawana marobhoti.txt faira roona chinongedzo kune saitimepu XML faira, rinogona kushandisa faira iri kuwana ese aripo maURL nezviwanikwa pasaiti. Murairo unotsanangurwa muchimiro:

Sitemap: https://yoursite.com/filesitemap.xml

Uyu mutemo unowanzoiswa pamagumo egwaro pasina kusungirirwa kune chaiyo Mushandisi-Agent uye inogadziriswa nemarobhoti ese pasina kusarudzika. Kana muridzi wesaiti akasashandisa sitemap.xml, hazvifanirwe kuwedzera mutemo.

Mienzaniso yeConfigured Robots.txt

Kumisikidza Robots.txt yeWordPress

Muchikamu chino, isu tichatarisa yakagadzirira-yakagadzirwa gadziriso yeWordPress. Isu tichaongorora kuvharidzira kupinda kune zvakavanzika data uye kubvumidza kupinda kune iwo makuru mapeji.

Semhinduro yakagadzirira, unogona kushandisa kodhi inotevera:

User-agent: *
# Block access to files containing confidential data
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php

# Allow access to the main site pages
Allow: /wp-content/uploads/
Allow: /sitemap.xml
Allow: /feed/
Allow: /trackback/
Allow: /comments/feed/
Allow: /category/*/*
Allow: /tag/*

# Prohibit the indexing of old versions of posts and parameterized queries to avoid content duplication or suboptimal indexing.
Disallow: /*?*
Disallow: /?s=*
Disallow: /?p=*
Disallow: /?page_id=*
Disallow: /?cat=*
Disallow: /?tag=*

# Include the sitemap (location needs to be replaced with your own)
Sitemap: http://yourdomain.com/sitemap.xml

Kunyangwe zvese zvinongedzo zvichiperekedzwa nemhinduro, ngationgororei zvakadzama mumhedziso.

1. Marobhoti haaratidze mafaera anonzwisisika uye madhairekitori.
2. Panguva imwecheteyo, marobhoti anobvumirwa kupinda kune mapeji makuru uye zviwanikwa zvesaiti.
3. kurambidzwa kunoiswa pakunongedza zvinyorwa zvekare zvezvinyorwa uye mibvunzo ine parameterized kudzivirira kudzokororwa kwemukati.
4. Nzvimbo yesaitimepu inoratidzirwa kune yakavandudzwa indexing.

Nokudaro, takatarisa muenzaniso wakakwana wegadziriro yakagadzirira, umo mamwe mafaira akajeka uye nzira dzakavanzwa kubva kune indexing, asi magwaro makuru anowanikwa.

Kusiyana neCMS dzakawanda dzakakurumbira kana masaiti akanyorwa netsika, WordPress ine akati wandei maplugins anofambisa kusika uye manejimendi eiyo robots.txt faira. Imwe yemhinduro dzakakurumbira dzechinangwa ichi Yoast SEO.

Kuti uiise, unoda:

1. Enda kune WordPress admin panel.
2. Muchikamu che "Plugins", sarudza "Wedzera Nyowani".
3. Tsvaga iyo "Yoast SEO" plugin uye uiise.
4. Shandisai plugin.

Kuti ugadzirise iro robots.txt faira, unofanira:

1. Enda kuchikamu che "SEO" mune iyo admin panel's side menu uye sarudza "General".
2. Enda kune "Tools" tab.
3. Dzvanya pa "Files". Pano uchaona mafaira akasiyana-siyana, kusanganisira robots.txt.
4. Isa mitemo inodiwa ye indexing zvinoenderana nezvaunoda.
5. Mushure mekuita shanduko kufaira, tinya bhatani "Chengetedza shanduko ku robots.txt".

Ziva kuti imwe neimwe robots.txt faira yekumisikidza yeWordPress yakasarudzika uye zvinoenderana nezvinodiwa uye maficha esaiti. Iko hakuna universal template ingakodzera zvese zviwanikwa pasina kusarudzika. Nekudaro, uyu muenzaniso uye kushandiswa kwema plugins kunogona kurerutsa basa.

Manual Setting yeRobots.txt

Saizvozvo, iwe unogona kuseta yako kumisikidzwa kwefaira kunyangwe pasina CMS yakagadzirira yesaiti. Mushandisi anodawo kuisa iyo robots.txt faira kumudzi wedhairekitori resaiti uye tsanangura mitemo inodiwa. Heino mumwe wemienzaniso, umo mirairo yese iripo inoratidzwa:

User-agent: *
Disallow: /admin/             # Prohibit access to the administrative panel
Disallow: /secret.html	      # Prohibit access to a specific file
Disallow: /*.pdf$	      # Prohibit indexing of certain file types
Disallow: /*?sort=	      # Prohibit indexing of certain URL parameters
Allow: /public/		      # Allow access to public pages
Sitemap: http://yourdomain.com/sitemap.xml # Include the sitemap

Maonero eRobots.txt File

Sechishandiso chekubatsira paunotarisa robots.txt faira yezvikanganiso, zvinokurudzirwa kushandisa mabasa epaIndaneti.

Chimbofunga muenzaniso we Yandex Webmaster service. Kuti utarise, iwe unofanirwa kuisa chinongedzo kune yako saiti mumunda unoenderana kana iyo faira yatoiswa kune server. Mushure meizvozvo, chishandiso pachacho chinotakura iyo faira gadziriso. Pane zvakare sarudzo yekuisa iyo yekumisikidza nemaoko:

Tevere, iwe unofanirwa kukumbira cheki uye kumirira mhinduro:

Mumuenzaniso wakapihwa, hapana zvikanganiso. Kana paine, sevhisi icharatidza nzvimbo dzine dambudziko uye nzira dzekudzigadzirisa.

mhedziso

Muchidimbu, takasimbisa kukosha kweiyo robots.txt faira pakudzora traffic panzvimbo. Isu takapa zano rekuti tingaimisa sei nemazvo kubata sei injini dzekutsvaga mapeji. Pamusoro peizvi, takatarisawo mienzaniso yemashandisiro ekushandisa faira iri nemazvo uye tikapa mirairo yekutarisa kuti ese magadzirirwo ari kushanda nemazvo.