Robots.txt

Maqaalkan, waxaan ku baari doonaa doorka muhiimka ah ee faylka robots.txt ee maaraynta taraafikada bogagga shabakadaha, ka doodo baahida jiritaankeeda, oo aan bixino talooyin loogu talagalay dejinta maareynta tusmaynta bogga waxtarka leh. Intaa waxaa dheer, waxaanu falanqeyn doonaa tusaalooyinka isticmaalka dardaaranka saxda ah ee ku jira faylka robots.txt waxaanan bixin doonaa hage ku saabsan sida loo hubiyo saxnaanta goobaha.

Waa maxay sababta Robots.txt loogu baahan yahay

Robots.txt waa fayl ku yaal server-ka goobta ee hagaha xididka. Waxay ku wargelinaysaa matoorada raadinta sida ay tahay inay u sawiraan waxa ku jira ilaha. Isticmaalka saxda ah ee faylkan wuxuu caawiyaa ka hortagga tusmooyinka boggaga aan loo baahnayn, wuxuu ilaaliyaa xogta sirta ah, wuxuuna hagaajin karaa waxtarka SEO-ka iyo muuqaalka goobta natiijooyinka raadinta. Qaabeynta robots.txt waxaa lagu sameeyaa dardaaranno, kaas oo aan sii eegi doono.

Dejinta Awaamiirta gudaha Robots.txt

Wakiilka Adeegsiga

Awaamiirta aasaasiga ah waxaa loo yaqaanaa Isticmaalaha-Wakiilka, halkaas oo aan u dejinay kelmad gaar ah oo loogu talagalay robots-yada. Marka la ogaado kelmaddan, robot-ku wuu fahmayaa in xeerka si gaar ah loogu talagalay.

Tixgeli tusaale isticmaalka Wakiilka-Isticmaalka ee faylka robots.txt:

User-Agent: *
Disallow: /private/

Tusaalahan ayaa tilmaamaya in dhammaan robots-ka raadinta (oo ay matasho calaamadda "*") waa in ay iska indhatiraan boggaga ku yaala /gaar ah/ tusaha.

Waa kuwan sida edbinta u eegayso aaladaha raadinta gaarka ah:

User-Agent: Googlebot
Disallow: /admin/

User-Agent: Bingbot
Disallow: /private/

Xaaladdan oo kale, the Googlebot robot raadinta waa in ay iska indhatiraan boggaga ku jira /Admin/ tusaha, halka binbot waa in la iska indhatiraa boggaga ku jira /gaar ah/ tusaha.

Diidmo

Diidmo u sheegaa robots-ka raadinta URL-yada ay ka boodaan ama aanay ku muujinayn shabakada. Dardaarankani waa mid faa'iido leh marka aad rabto inaad qariso xogta xasaasiga ah ama boggaga ka kooban tayada hoose si ay u muujiyaan matoorada raadinta. Haddii faylka robots.txt uu ka kooban yahay gelitaanka Diid: /directory/, ka dib robots waa loo diidi doonaa inay galaan waxa ku jira hagaha la cayimay. Tusaale ahaan,

User-agent: *
Disallow: /admin/

Qiimahani wuxuu tilmaamayaa taas dhammaan robots waa inay iska indhatiraan URL-yada ka bilaabmaya /Admin/. Si aad uga joojiso goobta oo dhan in ay ku tusmeeyaan robots kasta, u deji tusaha xididka sida qaanuun:

User-agent: *
Disallow: /

U oggolow

Qiimaha "Ogolow" wuxuu u dhaqmaa ka soo horjeeda "Disallow": waxay u ogolaataa robots raadinta inay galaan bog gaar ah ama hagaha, xitaa haddii awaamiirta kale ee faylka robots.txt ay mamnuucayaan gelitaanka.

Ka fiirso tusaale:

User-agent: *
Disallow: /admin/
Allow: /admin/login.html

Tusaalahan, waxa lagu caddeeyey in aan robots-yada loo oggolayn inay galaan /Admin/ tusaha, marka laga reebo /admin/login.html bogga, kaas oo diyaar u ah tusmaynta iyo iskaanka

Robots.txt iyo Khariidadda Goobta

Khariidadda bogga waa faylka XML oo ka kooban liiska URL-yada dhammaan bogagga iyo faylasha goobta kuwaas oo lagu tilmaami karo makiinadaha raadinta. Marka robot-ka raadinta uu galo faylka robots.txt oo uu arko isku xirka faylka XML ee khariidadda goobta, wuxuu isticmaali karaa faylkan si uu u helo dhammaan URL-yada iyo agabyada la heli karo ee goobta. Dardaaranka waxaa lagu qeexay qaabka:

Sitemap: https://yoursite.com/filesitemap.xml

Xeerkan waxaa inta badan la dhigaa dhamaadka dukumeentigu iyadoon lagu xidhin wakiil-Isticmaal gaar ah oo ay farsameeyaan dhammaan robots iyada oo aan laga reebin. Haddii mulkiilaha goobta uusan isticmaalin sitemap.xml, muhiim maaha in lagu daro qaanuunka.

Tusaalooyinka Robots La Habeeyay.txt

Dejinta Robots.txt ee WordPress

Qaybtan, waxaan tixgelin doonaa qaabeynta diyaarsan ee WordPress. Waxaan sahamin doonaa xannibaadda gelitaanka xogta sirta ah iyo u oggolaanshaha gelitaanka boggaga muhiimka ah.

Sida xal diyaar ah, waxaad isticmaali kartaa code soo socda:

User-agent: *
# Block access to files containing confidential data
Disallow: /cgi-bin
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php

# Allow access to the main site pages
Allow: /wp-content/uploads/
Allow: /sitemap.xml
Allow: /feed/
Allow: /trackback/
Allow: /comments/feed/
Allow: /category/*/*
Allow: /tag/*

# Prohibit the indexing of old versions of posts and parameterized queries to avoid content duplication or suboptimal indexing.
Disallow: /*?*
Disallow: /?s=*
Disallow: /?p=*
Disallow: /?page_id=*
Disallow: /?cat=*
Disallow: /?tag=*

# Include the sitemap (location needs to be replaced with your own)
Sitemap: http://yourdomain.com/sitemap.xml

Inkasta oo dhammaan awaamiirta ay la socdaan faallooyin, aan si qoto dheer u sii wadno gabagabada.

1. Robotyadu ma tusi doonaan faylasha xasaasiga ah iyo hagayaasha.
2. Isla mar ahaantaana, robots ayaa loo oggol yahay inay galaan boggaga ugu muhiimsan iyo ilaha goobta.
3. Mamnuucida waxa loo dejiyay tusmaynta qoraalada hore ee qoraalada iyo su'aalaha la qiyaasi karo si looga hortago nuqul ka mid ah qoraalka.
4. Halka ay ku taal khariidadda goobta waxa loo tilmaamay in la hagaajiyay.

Sidaa daraadeed, waxaanu tixgelinnay tusaale guud oo ah qaabeynta diyaarsan, kaas oo qaar ka mid ah faylasha xasaasiga ah iyo waddooyinka ay ka qarsoon yihiin tusmaynta, laakiin hagayaasha ugu muhiimsan waa la heli karaa.

Si ka duwan CMS badan oo caan ah ama goobaha sida gaarka ah loo qoray, WordPress waxa uu leeyahay dhowr plugins oo fududeeya abuuritaanka iyo maamulka faylka robots.txt. Mid ka mid ah xalalka caanka ah ee ujeedadan waa Yoast SEO.

Si aad u rakibto, waxaad u baahan tahay:

1. Tag guddiga maamulka WordPress.
2. Qaybta "Plugins", dooro "Kudar Cusub".
3. Soo hel plugin "Yoast SEO" oo ku dheji.
4. Furaha fiilada.

Si aad u tafatirto faylka robots.txt, waxaad u baahan tahay:

1. Tag qaybta "SEO" ee ku taal qaybta maamulka oo dooro "Guud".
2. Tag "Tools" tab.
3. Guji "Files". Halkan waxaad ku arki doontaa faylal kala duwan, oo ay ku jiraan robots.txt.
4. Geli xeerarka tusmaynta lagama maarmaanka ah sida waafaqsan shuruudahaaga.
5. Kadib samaynta isbedelada faylka, dhagsii badhanka "Save Change to robots.txt".

Ogsoonow in goob kasta oo faylka robots.txt ee WordPress uu yahay mid gaar ah waxayna kuxirantahay baahiyaha gaarka ah iyo astaamaha goobta. Ma jiro template caalami ah oo ku habboon dhammaan kheyraadka iyada oo aan laga reebin. Si kastaba ha ahaatee, tusaalahan iyo isticmaalka plugins ayaa si weyn u fududayn kara hawsha.

Dejinta gacanta ee Robots.txt

Sidoo kale, waxaad dejin kartaa qaabeynta faylkaaga xitaa haddii aysan jirin CMS diyaar u ah goobta. Isticmaaluhu sidoo kale wuxuu u baahan yahay inuu ku dhejiyo faylka robots.txt tusaha xididka ee goobta oo uu qeexo sharciyada lagama maarmaanka ah. Waa kan mid ka mid ah tusaalooyinka, kaas oo dhammaan awaamiirta la heli karo lagu tilmaamay:

User-agent: *
Disallow: /admin/             # Prohibit access to the administrative panel
Disallow: /secret.html	      # Prohibit access to a specific file
Disallow: /*.pdf$	      # Prohibit indexing of certain file types
Disallow: /*?sort=	      # Prohibit indexing of certain URL parameters
Allow: /public/		      # Allow access to public pages
Sitemap: http://yourdomain.com/sitemap.xml # Include the sitemap

Sida loo hubiyo Robots.txt File

Sida qalab caawiye ah markaad hubinayso faylka robots.txt khaladaadka, waxaa lagu talinayaa in la isticmaalo adeegyada internetka.

U fiirso tusaale ahaan Yandex Webmaster adeeg Si aad u hubiso, waxaad u baahan tahay inaad geliso isku xirka goobtaada goobta u dhiganta haddii faylka mar hore lagu dhejiyay server-ka. Taas ka dib, qalabka laftiisa ayaa ku shubi doona qaabeynta faylka. Waxa kale oo jira ikhtiyaar lagu galo qaabaynta gacanta:

Marka xigta, waxaad u baahan tahay inaad codsato jeeg oo aad sugto natiijooyinka:

Tusaalaha la soo bandhigay, ma jiraan wax khaladaad ah. Haddii ay jiraan, adeeggu wuxuu tusi doonaa meelaha dhibaatadu ka jirto iyo siyaabaha lagu hagaajinayo.

Ugu Dambeyn

Marka la soo koobo, waxaan xooga saarnay sida ay muhiimka u tahay faylka robots.txt ee lagu xakameynayo taraafikada goobta. Waxaan bixinay talo ku saabsan sida saxda ah ee loo dejiyo si loo maareeyo sida matoorada raadinta bogagga index. Intaa waxaa dheer, waxaan sidoo kale eegnay tusaalooyin ku saabsan sida saxda ah ee loo isticmaalo faylkan waxaanan siinay tilmaamo ku saabsan sida loo hubiyo in dhammaan goobaha ay si sax ah u shaqeynayaan.