What is Robots.txt File and How to Set It

As a blogger, maybe you are ever to heard the word of ‘robots.txt’. I believe, you surely confused when heard it. Well, I also confused when i heard it first time. But Now, i had know it and i will share and explain it to you as my best.

What is Robots.txt

Robot.txt file is a file with text(.txt) format that owned by each sites that online in internet. Any sites should have this file or the security of that site will broken. Robot.txt serves to control and regulate which the pages or directories will published in search engines. how it works is by put some codes inside it to control search engines to crawl or not the pages in 捷凯 that website. It surely to protect your system pages to known by people. Administrator pages is a page which abstinence to known by peoples or indexed by search engines. So, all of sites surely disallowing it administrator pages to indexed by search engines. Well, It is did by robot.txt. Robot.txt allow you to manage which the pages may crawled and indexed by search engines.
same as website, Blog also have robots.txt file. Only, robots.txt in a blog usually have set as default by the blogger platform. For blogspot blog, the default of the robots.txt is set like this :

User-agent: Mediapartners-Google

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://blogURL/feeds/posts/default?orderby=UPDATED

The meaning of the code above is the robots.txt file allow the google adsense crawler to crawl all pages of your site invariably, and allow all crawler to crawl all of your blog pages except the page with url prefix blogURL/search. The pages with this url prefix is labels and archives post pages.

How to Set Robots.txt

If you want to modify and set your robots.txt, firstly you must be careful because an error of your setting will make your website SERP on search engines disappear. You must have a clear purpose because it is not a little thing for your site/blog.
Okay, now I will tell you how to set it by right. You must know the meaning of each codes in robots.txt file above.

” user-agent : Mediapartners-Google ” : it is means that the codes which placed under it apply only with Mediapartners-Google crawler. Mediapartners-Google is google adsense crawler.

” Disallow: ” : It is means that no pages disallowed to crawl by crawler. By other word, allow the crawler to crawl all pages on the site.

” user-agent : * ” : it is means that the codes which placed under it is apply with all of search engines/crawler

” Disallow: /search ” : Disallow the crawler selected to crawl the url that have prefix blogurl/seach…

” Allow: / ” : Allow the crawler to crawl all of pages.

” Sitemap: http://blogURL/feeds/posts/default?orderby=UPDATED ” : this is sitemap of your site which submitted to crawled by crawler.

Not only the codes above can use in robots.txt. Just for the example, i want to blocked this post by robots.txt in order do not indexed by all of search engines. so i can do it by put this code.

user-agent : *
Disallow: /2012/11/what-is-robots-and-how-set-it.html

well, to block the root folders or directories(included it contents) in your site, you can do this :

user-agent : *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/

we can also set robots.txt in order to do not crawl and index a specific keyword. the example we want to block the word “people” to no-indexed by all search engines. we can set like this :

user-agent : *
Disallow: /search/*people

we can also set robots.txt to block the file in your site with specific format. The example we will block file with format php(.php), css(.css) and javascript(.js).

user-agent : *
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.css$

User Agent

In setting of your robots.txt, you can set which the crawler you allowed or disallowed to crawl. This is did by setting the user agent. Therefore, you must know the names of some famous crawlers/spiders. This is it lists :

Google adsense – Mediapartners-google
Google – Googlebot
Altavista – Scooter
Lycos – Lycos_Spider_(T-Rex)
Alltheweb – FAST-WebCrawler/
– Yahoo Slurp
MSN- Msnbot

*note :
-Any example i explain above is for , except the example for block folders/directories (it is for . Each sites have different set of robots.txt. It is because each sites have different directories too. The setting robots.txt of wordpress blog surely will different with blogspot blog, but the rules of codes are same.
-If you have a sites which no use blogspot platform. you must add robots.txt file and locate it on main root of your site.

well, that’s all. Still any questions ? don’t hesitate to ask me. Now, you have know and can set your robots.txt file of your site. i waiting for your response. Hopefully useful for you. Keep happy blogging guys 🙂

side effects of viagra lexapro coupons lexapro vs celexa cheap levitra azithromycine 250 mg lexapro generic name azithromycin three times a week cialis pricing what is acyclovir used for viagra nebenwirkungen cialis 5mg levitra 20 mg cost walmart azithromycin vs erythromycin ciprofloxacin side effects in women ciprofloxacin ophthalmic solution generic viagra buy viagra pfizer viagra coupons from pfizer cipro uses cialis dosage strengths tamoxifen and foods to avoid finasteride 1mg azithromycin 500mg lexapro weight gain cialis patent expiration date extended how to take cialis cialis dosage what works better than viagra azithromycin for uti how long for lexapro to work acyclovir cream cipro antibiotic azithromycin eye drops stopping lexapro azithromycin 250mg cost of cialis viagra without a doctor prescription usa levitra bayer 20mg meilleur prix levitra prezzo in farmacia lexapro side effects men how long does viagra last escitalopram lexapro 10 mg buying viagra finasteride vs dutasteride cialas cialis 5 mg azithromycin for pneumonia escitalopram effets secondaires azithromycine eg 500 mg bijsluiter what does azithromycin treat azithromycin class ciprofloxacin 500mg antibiotics acyclovir 800 mg for shingles viagra on line no prec ciprofloxacin used for ciprofloxacin 500 mg tabletten levitra vs viagra ciprofloxacin bnf sex viagra for women first few days on lexapro cialis patent expiration 2017 cipro side effects in elderly buy generic levitra viagra vs cialis wholesale cialis ciprofloxacin dose cialis generic tadalafil side effects of azithromycin zoloft vs lexapro azithromycin 250mg tablets 6 pack ciprofloxacin drops side effects of lexapro what is escitalopram ciprofloxacin 500 mg for urinary tract infection what does viagra do viagra en ligne livraison 24h side effects of cialis lexapro 20mg levitra 20 mg canadian viagra cialis pharmacy prices ciprofloxacin nebenwirkungen escitalopram oxalate tamoxifen cost lexapro dosages cilias cialis 20 mg female viagra cipro for uti liquid cialis cipro dangers lexapro and wellbutrin combination therapy acyclovir side effects cialis online viagra receptfritt tamoxifen side effects is azithromycin penicillin finasteride 5mg side effects mayo clinic finasteride 5 mg tab cheap cialis side effects of finasteride 5mg buy cialis buy levitra generic cialis tamoxifen therapy cipro 500 cialis 5mg daily tamoxifen nebenwirkungen finasteride and enlarged prostate cialis coupon print viagra ohne rezept aus deutschland escitalopram 10 mg para que sirve side effects of lexapro 10 mg side effects lexapro finasteride results free viagra viagra natural escitalopram side effects weaning off lexapro cialis vs viagra azithromycin 5 day dose pack cialis cheap viagra samples cialis medication nolvadex viagra 100 mg viagra 100mg what is tamoxifen used for zovirax cream cialis coupons printable cialis for women cheap viagra lexapro overdose what infections does ciprofloxacin treat cialis 20mg directions lexapro dosing cialis from canada azithromycin z pak side effects of acyclovir cialis effects what is tamoxifen finasteride long term effects 9 levitra at walmart lexapro side effects cialis for daily use what is azithromycin for viagra for men escitalopram 10 mg ciprofloxacin 500 mg acyclovir for cold sores azithromycin dosage how to wean off lexapro without withdrawals zithromax propecia acyclovir dosing finasteride efectos secundarios what is cipro usually prescribed for finasteride for hair loss viagra soft azithromycin liquid cealis how does lexapro work canada cialis what is ciprofloxacin prescribed for azithromycin side effects viagra wikipedia cialis sample viagra 50mg ciprofloxacin hcl equipe argentine viagra generic viagra 100mg cialis reviews finasteride side effects in men thuoc azithromycin tamoxifen and bone pain order viagra cialis coupon 20 mg finasteride effets secondaires 200 cialis coupon what is ciprofloxacin acyclovir medication cialis canadian pharmacy withdrawal from lexapro zovirax pills herbal viagra zithromax dosage acyclovir 200mg viagra tesco buy levitra online cialis coupons online cialis acyclovir 400mg finasteride 5mg levitra 20mg cost per pill cyalis tamoxifen tamoxifen side effects in women over 55 cialis on line viagra for sale side effects of azithromycin 250 mg acyclovir dose is azithromycin viagra tablet is ciprofloxacin a penicillin tamoxifen langzeitfolgen viagra wiki viagra sex viagra alternatives cialis samples overnight ciprofloxacin hcl 500 mg lexapro 10mg what is cipro what is lexapro used for generic for lexapro viagra bestellen cipro antibiotics generic cialis tadalafil lexapro withdrawal sophia viagra how does acyclovir work levitra dosage ciprofloxacin 250mg viagra dosage viagra uk how to get off lexapro viagra ohne rezept auf rechnung azithromycin 250 effects of stopping finasteride ciprofloxacin hcl 500 mg tab lexapro side effects in women levitra vs viagra for hardness levitra rezeptfrei deutschland lexapro and weight gain finasteride 5mg tab what is lexapro for finasteride 5mg side effects cialis canada viamedic cialis viagra 100 where to buy viagra escitalopram 20 mg acyclovir ointment 5 cialis 30 day sample cialis pills for sale azithromycin without a doctor’s prescription what does finasteride do viagra pill what is cialis what is acyclovir erectile dysfunction cialis ciprofloxacin side effects ciprofloxacin hydrochloride viagra for sale uk vardenafil 20mg levitra 20 mg von bayer is tamoxifen chemotherapy how does viagra work price cialis ciprofloxacin side effects after age 60 cialis daily tamoxifen lawsuits azithromycin 250 mg treatment ciprofloxacin spc viagra medicine acyclovir 800 mg 5 times a day cialis website viagra activate what is ciprofloxacin used for levitra erfahrungen cialis discount cialis prices cialis 10 mg wellbutrin and lexapro how to use viagra cialis canadian cialis viagra vs cialis daily cialis viagra on line levitra 10 mg prezzo cialis generic cipro dosage how does cialis work cialis savings card lexapro manufacturer website acyclovir discount cialis cialis pills side effects of finasteride ciprofloxacin 250 mg ciprofloxacin drug class viagra coupon what is cipro used for acyclovir 800mg what is azithromycin used for tamoxifen interaction online viagra lexapro benefits for women cialis samples lexapro bula viagra viagra vs cialis vs levitra generic levitra propecia side effects azithromycin pediatric dosing what does ciprofloxacin 500mg treat cialis free trial natural viagra ciprofloxacin 500mg tablets finasteride tablets viagra cost women viagra viagra sans ordonnance lowest cialis prices viagra costs finasteride permanent side effects azithromycin 500 cialis dosage recommendations what is finasteride generic cialis at walmart celexa vs lexapro azithromycin and alcohol cialis without a doctor’s prescription discount viagra lexapro vs wellbutrin cialis trial cialis for bph azithromycin tablets lexapro medication does lexapro cause weight gain cialis 30 day trial coupon ciprofloxacin dosage how long for azithromycin to take effect cialis coupons 2017 viagra cialis post finasteride syndrome what is finasteride used for utilisation viagra ciprofloxacin for dogs what is viagra azithromycin cost viagra canada zovirax acyclovir ointment tamoxifen side effects in women finasteride hair loss levitra ciprofloxacin hydrochloride ophthalmic escitalopram 10mg finasteride buy cialis online ciprofloxacin 500 cialis vs viagra zithromax antibiotic cialis or viagra zithromax 500 mg azithromycin pregnancy viagra pillen kruidvat viagra tablets levitra online cialis for sale viagra online levitra 20mg viagra alternative azithromycin 500 erfahrungen viagra without a doctor prescription cipro drug class cialis price ciprofloxacin over the counter viagra generic levitra vardenafil 20mg levitra vs viagra levitra coupons finasteride reviews ciprofloxacin 500mg antibiotics side effects cialis coupons from manufacturer ciprofloxacin class cialis for men finasteride medication what is azithromycin lexapro viagra kaufen side effects of ciprofloxacin viagra side effects tamoxifen and weight gain side effects of lexapro in women acyclovir dosage for cold sores ciprofloxacin hcl 500 mg side effects cialis copay card azithromycin 250 mg cialis professional finasteride for women cialis otc how to wean off lexapro generic lexapro azithromycin 500 mg azithromycin drug class zithromax z-pak azithromycine lexapro para que sirve antibiotic ciprofloxacin ciprofloxacin eye drops cialis tadalafil viagra effects cipro and tendonitis tamoxifen citrate viagra pills is lexapro a controlled substance cipro side effects finasteride 5 mg viagra prices cost of viagra common side effects of ciprofloxacin 500 mg azithromycin pregnancy category viagra generic cialis patent expiration viagra from canada canada viagra what is acyclovir for what bacteria does ciprofloxacin kill cialis side effects ciprofloxacin uses levitra prices buy generic viagra effects of stopping lexapro suddenly acyclovir 400 mg cipro buy viagra online viagra substitute azithromycin for strep throat lexapro reviews cialis voucher viagra generico lexapro and weight loss ciprofloxacin 500mg free cialis zovirax for cold sores zovirax ointment viagra vs cialis vs levitra ciprofloxacin coverage azithromycin coverage lexapro for anxiety zovirax dosage viagra coupons lexapro dosage escitalopram nebenwirkungen azithromycin cialis generic availability acyclovir vs valacyclovir cialis 20mg acyclovir dosage lexapro generic levitra 20 mg precio farmacia acyclovir for shingles levitra generic what does ciprofloxacin treat lexapro vs zoloft side effects of tamoxifen pfizer viagra levitra coupon how much does cialis cost lexapro and weight gain in women generic for cialis side effects of ciprofloxacin 500 mg cialis coupon acyclovir 800 mg finasteride dosage when will cialis go generic lexapro withdrawal symptoms what is lexapro cialis alternative azithromycin tablet levitra without a doctor prescription finasteride side effects viagra for women viagra 100mg tablets retail price lexapro side effects in men azithromycin medscape tamoxifen bijwerkingen best price viagra ciprofloxacin dosierung lexapro 5mg viagra single packs cialis cost vardenafil vs viagra azithromycin uses revatio vs viagra viagra price cialis testimonials

1 Comment

  1. Wow !!! I’ve been wondering what the Robot.txt means but i never knew.. Thanks so much for this post, i hope it will help my Technology Blog to rank better in Search Results.

Leave a Reply

Your email address will not be published. Required fields are marked *