Tuesday, July 28, 2009

iostat: (r/s + w/s) * svctm = %util on Linux

iostat -x is very useful to check disk i/o activities. Sometimes it is said that "check %util is less than 100%" or "check svctm is less than 50ms", but please do not fully trust these numbers. For example, the following two cases (DBT-2 load on MySQL) used same disks (two HDD disks, RAID1) and reached almost 100% util, but performance numbers were very different (no.2 was about twice as fast as no.1).
# iostat -xm 10
avg-cpu: %user %nice %system %iowait %steal %idle
21.16 0.00 6.14 29.77 0.00 42.93

Device: rqm/s wrqm/s r/s w/s rMB/s wMB/s
sdb 2.60 389.01 283.12 47.35 4.86 2.19
avgrq-sz avgqu-sz await svctm %util
43.67 4.89 14.76 3.02 99.83

# iostat -xm 10
avg-cpu: %user %nice %system %iowait %steal %idle
40.03 0.00 16.51 16.52 0.00 26.94

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
sdb 6.39 368.53 543.06 490.41 6.71 3.90
avgrq-sz avgqu-sz await svctm %util
21.02 3.29 3.20 0.90 92.66

100% util does not mean disks can not be faster anymore. For example, command queuing (TCQ/NCQ) or battery backed up write cache can often boosts performance significantly. For random i/o oriented applications(in most cases), I pay attention to r/s and w/s. r/s is the number of read requests that were issued to the device per second. w/s is the number of write requests that were issued to the device per second (copied from man). r/s + w/s is the total number of i/o requests per second (IOPS) so it is easier to check whether disks work as expected or not. For example, a few thousands of IOPS can be expected on single Intel SSD drive. For sequential i/o operations, r/s and w/s can be significantly affected by Linux parameters such as max_sectors_kb even though throughput is not different, so I check different iostat status variables such as rrqm/s, rMB/s.

What about svctm? Actually Linux's iostat calculates svctm automatically from r/s, w/s and %util. Here is an excerpt from iostat.c .

...
nr_ios = sdev.rd_ios + sdev.wr_ios;
tput = ((double) nr_ios) * HZ / itv;
util = ((double) sdev.tot_ticks) / itv * HZ;
svctm = tput ? util / tput : 0.0;
...
/* rrq/s wrq/s r/s w/s rsec wsec rkB wkB rqsz qusz await svctm %util */
printf(" %6.2f %6.2f %5.2f %5.2f %7.2f %7.2f %8.2f %8.2f %8.2f %8.2f %7.2f %6.2f %6.2f\n",
((double) sdev.rd_merges) / itv * HZ,
((double) sdev.wr_merges) / itv * HZ,
((double) sdev.rd_ios) / itv * HZ,
((double) sdev.wr_ios) / itv * HZ,
...

The latter means the following.
r/s = sdev.rd_ios / itv * HZ
w/s = sdev.wr_ios / itv * HZ

The former means the following.
svctm = util / ((sdev.rd_ios + sdev.wr_ios) * HZ / itv)

If %util is 100%, svctm is just 1 / (r/s + w/s) seconds, 1000/(r/s+w/s) milliseconds. This is an inverse number of IOPS. In other words, svctm * (r/s+w/s) is always 1000 if %util is 100%. So checking svctm is practically as same as checking r/s and w/s (as long as %util is close to 100%). The latter (IOPS) is much easier, isn't it?

61 comments:

PaulM said...

So the take home message is:

Use iostat -xm, look at svctm when %util is near 100%. Smaller values is better as smaller IOPS is better :)

If you look at the two outputs you produced the 2nd set clearly shows this.

Thanks for the post. One of those nice posts which explain some commonly used tools on unix in more detail

Have Fun

famzah said...

This is so not accurate :)

"%util" is the total time the device spent in doing I/O (sdev.tot_ticks) divided by the wall clock time (itv). This gives us an idea about how busy a device is.

Example: If we are doing measurements every 1000 ms and for these 1000 ms the device spent 950 ms in doing I/O, then %util is 95%. It is very likely that this device cannot do any faster with the *current* I/O load.

"svctm" is the average service time for I/O requests. This means how much time the device spent on average for every I/O operation. This is the hardware time an I/O operation took to finish.

"svctm" is equal to the total time the device spent in doing I/O (sdev.tot_ticks) divided by the total number of I/O operations (nr_ios). Every device has its mechanical characteristics on how quickly it can do an I/O operation.

For example, a typical hard disk device performs an I/O operation in less than 5 ms. If you have high values for "svctm", this usually means that your device performs poorly in regards to hardware and probably needs replacement.

---

Note that all the above applies for the devices (ex. /dev/sda), and not for the partitions (ex. /dev/sda2). As you can see in "Documentation/iostats.txt" there are also statistics for the partitions but a discussion about them would require even more explanations.

Yoshinori Matsunobu said...

famzah,

If disk configurations are not optimal, disks might reach util=100% very quickly, but that is not an uppermost limit. Check two iostat results in my blog entry again. The difference is only "whether write cache on disks is enabled or not". People might decide to purchase additional H/W components when seeing the first iostat result (because %util is close to 100% and svctm seems small enough(3.02)), but in this case this is a wrong decision. Checking multiple iostat parameters (IOPS + %util, or svctm + %util) and understanding typical IOPS in database applications (sometimes IOPS on HDD can be high thanks to write cache and command queuing) are important and this is what I want to mention in this post.

famzah said...

Yoshinori,

I fully agree that if %util=100%, it doesn't always mean that the device cannot perform faster. It means that it cannot perform faster with the current "configuration" and "load".

Tweaks like enabling the cache can surely increase the performance of a device. This does not make %util less trustful though :) And yes, you should check at least IOPS + %util to see if you reached your hardware limits or there is some configuration issue.

I however still cannot grasp your idea about "svctm"... We should check "svctm" at any time, not only when %util=100%. And regardless of the device %util or IOPS load, "svctm" should always be below a certain value which is hardware dependent (like below 5 ms for most hard disks).

P.S. I do not want to start a war here, I'm just commenting on the semantics of "%util" and "svctm".

Yoshinori Matsunobu said...

famzah,

Yes, I agree with that checking svctm is a good practice. svctm might be high if IOPS is low, even though %util is low(svctm = %util / IOPS so low IOPS makes svctm high). http://www.itworld.com/UIR961001perf is interesting though I have not encountered this on Linux yet.

famzah said...

I haven't encountered this recently either, and I'm currently monitoring the "svctm" values of more than 230 Linux servers.

This seems like a (rare) bug, not a rule. In any case when there is a high "svctm", people should determine the following:
* is my device failing (by examining the IOPS + util + benchmarks)
* am I encountering the iostat bug (in 99.9% it's the first case, a hardware failure)

My recent experience shows that there is no way that "svctm" is higher than the expected maximum hardware service time and this is normal :)

Anyway, all is clear now and I have no further comments. Thanks for the article, it helps people understand "%util" and "svctm" better.

shallu said...

Here is a great chance to drive a large number of visitors to your blogs and websites for free.

Submit your websites, blogs, videos to http://www.zillionsb.com and get 1000s of targeted visitors everyday for free. It also helps your websites/blogs gain valuable backlinks.

Let the other bloggers cast their votes to push your posts up for a greater visibility. Enjoy a free, huge traffic to your sites.

Thanks

Sara
http://www.zillionsb.com

Sinus headache said...

Your post really grabbed my attention and interest for the reason that the content is not just informative but also simple yet meaningful.

Natural laxative foods Stomach gas Hard stool treatment Health benefits of almonds White spots on skin
Cure ear infection Under eye wrinkles Baby ear infection Prevent prostate cancer Prevent prostate cancer Lemon tea healh benefits Lemon tea healh benefits Prevent candida yeast infection Prevent candida yeast infection Health benefits of papaya Health benefits of papaya Benefits of peppermint oil Benefits of peppermint oil Stop hiccups Stop hiccups Water and weight loss Water and weight loss Wart Removal Wart Removal Health benefits of oatmeal

android tablet said...

Good post. Very impressive. Thanks for sharing.

RS Gold said...

If disk configurations are not optimal, disks may achieve util=100% extremely quickly, but that is not an uppermost limit. examine two iostat final results in my weblog entry again. The big difference is only "whether create cache on disks is enabled or not". men and women may choose to buy extra H/W elements when seeing the primary iostat outcome (because %util is near to 100% and svctm looks little enough(3.02)), but on this circumstance this is mainly a incorrect decision. examining several iostat parameters (IOPS + %util, or svctm + %util) and knowing common IOPS in databases applications (sometimes IOPS on HDD might be higher many thanks to create cache and command queuing) are necessary and this really is what I desire to mention on this post.

WOW Gold is always necessary to warcraft,like we Buy WOW Items to improve our account!!But we usually can't find a good place to buy WOW Items Gold ,Cheap WOW Items and no safe online shope Buy WOW Gold, it not like some game like tera,players can Buy Tera Gold oe even Cheap Tera Gold easily on some sites!

hermes birkin said...

Thank you for another essential article. Where else could anyone get that kind of information in such a complete way of writing? I have a presentation incoming week, and I am on the lookout for such information.
Hermes replica
hermes birkin replica

Runescape Gold said...

I would have the name of your blog Dreamland! While Santa knocks at our door just once a year, you blog is open all year - wow!

http://www.buylovejewelry.com/
http://gamepartygogo.com/

Anonymous said...

In the men's eyes, Omega hermes wallets is made of steel; in a woman's eyes, Omega is

made of water. None of Idol Replicas watche them are wrong, because gucci sunglasses they see

different blue dial replicas Breitling replica watches. Omega goes beyond the meaning of

time itself, which represents the "nobility" and "classic".

Happytera said...

The online gaming Tera Gold community is an extremely popular area for those who experience a true rush when playing Tera Account video games. In addition to this, many individuals have discovered that there are numerous social advantages to participating in online gaming communities. Businesses have made millions of dollars by sponsoring this online gaming community. There are many tera online gold online gaming types.

gamefan said...

Thanks for providing such useful information. I must say i many thanks for professional approach. I have to appreciate the efforts you have made on paper this post. I hope exactly the same best product within you in the foreseeable future in addition.
http://www.4rsgold.com/christmas/active.html
http://www.fzf.com/christmas/christmas.html

film art said...

This is a wonderful content. I will bookmark this site and visit again. It is very informative. Thanks for sharing.
direktorijum clanaka

lace wedding dress said...

A holy church, rings, a bouquet, a 3-floor wedding cake, lace wedding dress champagne, and the moving promise “ I do”, make up the happiest moment in one’s life. Then it is the wedding.
lace wedding gowns
A gorgeous wedding dress has been inside a girl’s dream since she was five years old. It is widely said that the bride is the most beautiful woman all over the world.
Bridal gowns vary in different parts of the world. For example, cheap wedding dresses plus sizein traditional Chinese culture, the color red is regarded for centuries as the symbol of good luck; while in the Occident, wedding dresses 2012 a full-length white wedding dress is always associated with romance and sanctity. But, with the globalization, wedding dresses tend to be alike in the whole world.

get girlfriend give oral sex said...

Excellent post. I want to thank you for this informative read. I will bookmark this site and visit again...

ambersbridal said...

A holy church, rings, a bouquet, a 3-floor wedding cake, lace wedding dress
champagne, and the moving promise “ I do”, make up the happiest moment in one’s life. Then it is the wedding.
lace wedding gowns
A gorgeous wedding dress has been inside a girl’s dream since she was five years old. It is widely said that the bride is the most beautiful woman all over the world.
Bridal gowns vary in different parts of the world. For example, cheap wedding dresses plus sizein traditional Chinese culture, the color red is regarded for centuries as the symbol of good luck; while in the Occident, wedding dresses 2012 a full-length white wedding dress is always associated with romance and sanctity. But, with the globalization, wedding dresses tend to be alike in the whole world.

Knjigovodstvena agencija Beograd said...

Excellent post, really. I like how do you solve this problem. Well done, mate!

Anonymous said...

Great post, man! Well done!
Ekologija
Kese za butik
recepti
advokatska kancelarija

Cheap RS Gold said...

Thanks for finding the time to discuss this, I am strongly about it and love learning more about this topic. If possible, when you gain expertise, does one mind updating your blog site to comprehend information? It is very of great help for me.
http://maplestoryuna.blogspot.com/2012/01/you-can-start-designing-by-using.html
http://wowgold.eklablog.com/tyra-banks-and-heidi-klum-two-popular-female-celebrities-a30683683
http://blog.palungjit.com/wowgolduna/stella-mccartney-designed-for-chloe-from-1997-to-2001.html
http://wowgold.allmyblog.com/33-westwood-s-anglomania-collection-is-a-more-casual.html
http://wowgolduna.blogturk.org/54464/Design+the+clothing+for+your+fashion+makeover+doll.html
http://www.mojatu.com/wowgolduna/blog/try-the-design-on-your-virtual-doll/

Bg steel said...

Excellent post, man! Well done!
Knjigovodstvena agencija

arabic letters said...

I want to thank you for this informative read. Loved the whole article! Thanks for sharing.
Does milk make you taller

Anonymous said...

I like this informative post, thanks.
tipovi

Samobor said...

Thank you for another essential article.

Croatia foto said...

Thank you for another good article.

Anonymous said...

Great and useful post, thanks.

http://plasticnekese.weebly.com/
http://ekologija12.blogspot.com/
http://celik1.blogspot.com/
http://knjigovodstvo1.blogspot.com/
http://www.squidoo.com/celici

Anonymous said...

Great post, man.

Plasticne kese
Ekologija
Celik
Knjigovodstvo
Celici

Anonymous said...

Incom Knjigovodstvo
Ratka Mitrovića 45
11030 Čukarica, Beograd



Tel: 011/2511-201
011/2511-404

i need a blowjob said...

I want to thank you for this informative read. Loved the whole article! Thanks for sharing.
jacks blowjob persuasion
how to get a girl to suck your dick
get girlfriend to deepthroat
convince girlfriend facial
how to get more blow jobs
i need a blowjob
wife wont suck my cock
make swallow cum
jacks blowjob persuasion review

seagrass baskets said...

This is a wonderful content. I will bookmark this site and visit again. It is very informative. Thanks for sharing.
restorani u beogradu

Maria Arredondo said...

Ability may get you to the top, but it takes character to keep you there, RuneScape Gold to change your future, alter your decisions today, RS Gold. Life is not measured by the number of breaths we take, but by the moments that take our breath away RS2 Gold.

Maria Arredondo said...

Would you write to please D3 Gold just yourself? Or others? Or yourself by writing for others Cheap D3 Gold? It takes strength to do what must be done when the work is unpleasant and uncomfortable D3 Gold Sale, And of what would you write: Of love? Hate? Fun? Misery? Life? Death?Nothing Everything?

buyGuildwars2gold said...

Do not pray for tasks equal to your powers.Pray for powers equal to Buy rs gold your tasks.Then the doing of work shall be no miracle, but you shall be the miracle, Fear not that the life shall come to an end, but rather fear that Runescape Money it shall never have a beginning,. At twenty years of age Runescape2 Money., the will reigns; at thirty, the wit; and at forty, the judgment.

buyGuildwars2gold said...

Would you write to please D3 Gold just yourself? Or others? Or yourself by writing for others Cheap D3 Gold? It takes strength to do what must be done when the work is unpleasant and uncomfortable D3 Gold Sale, And of what would you write: Of love? Hate? Fun? Misery? Life? Death?Nothing Everything?

WLKgames said...

Nowadays, WOW Rookie will not help you improve your game or your rs gold character's performance. So you should train them to be the excellent WOW Rookie. They suggest that won't help you become a better player; practically, we encourage you to guard against the urge wow gold.

Momo Wu said...

Very significant article for us, I think the representation of this article is actually superb one. This is my first visit to your site PTFE Rod

Vivian Salvatore said...

Guild Wars 2 Gold changes the course of the history, and the tasks anyone confront. Within the additional lifetime of the D3 Gold Online you may be with numerous tale objectives continuously facing the decision, the following objective may carry on consequently Gw2field.

Cahaya Mandiri said...

Great Post. I have not been visiting the site recently. Took a visit again and there were some great comments on the site. Excellent post. Keep up the good work.
tips cara agar cepat hamil l CARA BELAJAR BAHASA INGGRIS l the best acne treatment l how to lose weight fast easy
margahayuland l BISNIS ONLINE l tips cepat hamil l how to get rid of acne home remedies l
home remedies for acne l how to cure acne fast l
baju batik modern l toko sepatu online l grosir jam tangan online l
jual jam tangan l toko jam tangan murah peluang usaha online l is acne no more for you l how to get rid of acne naturally
how to clear acne l cure acne naturally
best natural remedies for acne l acne no more l tempat belajar bisnis online
peluang usaha rumahan l cara mendapatkan uang dari internet
makanan sehat agar cepat hamil l penyebab tidak bisa hamil lcara agar cepat hamil


smith rose said...

Adobe Acrobat 9 Pro Extended Download includes all the features and functionality of Acrobat 9 Pro, plus the ability to unify the widest range of content in a PDF Portfolio, create interactive presentations with Adobe Presenter software, easily convert and share video in PDF, create PDF maps, convert virtually any 2D and 3D designs to PDF, and enjoy expanded 3D capabilities with the new Adobe 3D Reviewer.
The Timeline view in Microsoft Project 2010 Download can be copied and pasted in an e-mail message or in any Office application for a quick view of Project progress along a graphic timeline.

smith rose said...

Visio Professional 2010 Download provides advanced diagramming tools to help you simplify complexity with intuitive and professional-looking diagrams, dynamic and data-driven visuals, and new ways to share via a browser in real time.
With modern, pre-drawn shapes, intelligent templates, and sample drawings, Microsoft Visio 2010 Download offers a wide variety of options to meet your diagramming needs for IT, business, process management, and more.

smith rose said...

Project 2010 Download includes a timeline view that is automatically displayed above other views, showing a concise overview of the entire schedule. You can add tasks to the timeline and even print it for an attractive summary report of the entire project. Or you can paste it into an e-mail for an instant report with no fuss.
Adobe Acrobat 9 Pro Extended Download includes all the features and functionality of Acrobat 9 Pro, plus the ability to unify the widest range of content in a PDF Portfolio, create interactive presentations with Adobe Presenter software, easily convert and share video in PDF, create PDF maps, convert virtually any 2D and 3D designs to PDF, and enjoy expanded 3D capabilities with the new Adobe 3D Reviewer.

smith rose said...

With Microsoft Project Professional 2010 Download, you can add Microsoft Project Server 2010 to realise unified project and program management.
This document introduces the new features of Project Standard 2010 and Project Professional 2010.

Momo Wu said...

It is a pleasure going through your post. I have bookmarked you to check out new stuff from your side forgot Windows password

smith rose said...

Visio Standard 2010 Download offers modern and intuitive diagramming tools to transform complex ideas into aha moments and get everyone on the same page with less time and effort. A diverse set of pre-drawn shapes, pictures, and templates, and new automatic drawing tools make visualization easier than ever.

Grosir Pulsa Online said...

Wonderful things coming this time, and always raise your hands to the sky! regards - daftar makanan tradisional , kue dr tepung beras , jual oleh oleh padang , cemilan untuk diet

Momo Wu said...

I am not sure where you are getting your info, but good topic. I needs to spend some time learning much more or understanding more. Thanks for magnificent information I was looking for this information for my mission.The Best Watches Review

sadia sulaman said...

Excellent read, Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style.www.thepsychicchildren.co.uk |

www.siamfishingtours.co.uk |

www.dogsoncanvas.co.uk |

www.stable-barrow.co.uk |

www.gigidagostino.co.uk |

www.voidaudio.co.uk |

www.sharinghealth.co.uk |

www.axispropertyinvestment.co.uk |

www.lifeinsport.co.uk |

www.buckbyblinds.co.uk |

devon broad said...

And additionally here is that the smartest thing of, your own initial check of one's day and Fitting loan is certainly fascination freed from CHARGE!!! That’s applicable. you will not ought to pay one dime related check cashing to fascination on your terribly initial check. The day and Fitting loan are going to be thought of one among every of our hottest money merchandise, because it is so hassle-free and additionally accommodating.

anika hansika said...

Payday cash changes, Salaryday what is a ton of Dimensions Usecured bank credits, in conjunction with auto Strategy Usecured bank propels. we've got an inclination to watch out for tend to target supporting you choose to buy for payday loans the comfortable thing likewise as thing or organization completely energize satisfy your current conservative needs.

panca-samudera said...

The article posted was very informative and useful
thanks for sharing.
jaring futsal , jaring futsal murah , jual jaring futsal , toko jaring futsal , agen jaring futsal , jaring golf , jual jaring golf , jaring golf murah ,
jaring polynet , tangga darurat , jaring truk , jaring cargo , jaring outbound , jual rumput futsal murah , jual rumput sintetis murah ,
tali tambang , cargo net , jaring tanaman , jaring kassa , jaring proyek , jaring bangunan , jaring gedung , jaring pengaman proyek ,
jaring pengaman bangunan , jaring pengaman gedung , jaring peneduh , jaring waring , kasa hijau , tangga darurat , jaring gawang futsal , jual jaring gawang futsal murah , jaring peneduh , jaring truk , tali tambang nylon

Minh Vương said...

Công ty vệ sinh 5 Sạch
Dịch vụ giặt thảm tại Hà Nội
Dịch vụ giặt ghế sofa tại Hà Nội
Vệ sinh nhà sau xây dựng

fikri ilham said...

Jual obat herbal de nature asli untuk berbagai penyakit kelamin dan sebagainya
Obat Wasir Herbal
Obat Wasir Herbal Mujarab
Obat Wasir Herbal Ampuh
Obat Wasir Herbal Pada Pria
Obat Wasir Herbal Di Apotik
Cara Mengobati Wasir Stadium 4
Cara Mengobati Wasir Stadium 4 Tanpa Operasi
Cara Mengatasi Wasir Stadium 4
Cara Penyembuhan Wasir Stadium 4
Cara Mengobati Ambeien Stadium 4 Secara Alami
Cara Pengobatan Wasir Stadium 4
Cara Pengobatan Ambeien Stadium 4
Cara Mengobati Wasir Stadium 3 Dan 4
Cara Mengobati Wasir Stadium 3
Cara Mengobati Wasir Stadium 3 Secara Alami
Obat Wasir Herbal Apotik
Obat Wasir Herbal Untuk Ibu Hamil
Obat Wasir Herbal Terdaftar Bpom
Obat Wasir Herbal Untuk Ibu Menyusui
Obat Wasir Herbal Bandung
Cara Mengobati Wasir Stadium 3 Dan 4
Cara Mengatasi Wasir Stadium 3
Cara Mengobati Ambeien Stadium 3 Tanpa Operasi
Cara Menyembuhkan Wasir Stadium 3
Cara Mengatasi Ambeien Stadium 3
Cara Alami Mengobati Wasir Stadium 3
Cara Mengobati Ambeien Stadium 3 Secara Alami
Obat Wasir Herbal Manjur
Obat Wasir Herbal Semarang
Obat Wasir Herbalife
Obat Wasir Herbal Jogja
Obat Herbal Wasir Berdarah
Obat Herbal Wasir Dalam
Obat Herbal Wasir Akut
Obat Herbal Wasir Stadium 4
Obat Herbal Wasir Terpercaya
Obat Herbal Wasir Stadium 3
Obat Herbal Wasir Internal
Obat Wasir Herbal Alami

Dr Purva Pius said...

Hello Everybody,
My name is Mrs Sharon Sim. I live in Singapore and i am a happy woman today? and i told my self that any lender that rescue my family from our poor situation, i will refer any person that is looking for loan to him, he gave me happiness to me and my family, i was in need of a loan of S$250,000.00 to start my life all over as i am a single mother with 3 kids I met this honest and GOD fearing man loan lender that help me with a loan of S$250,000.00 SG. Dollar, he is a GOD fearing man, if you are in need of loan and you will pay back the loan please contact him tell him that is Mrs Sharon, that refer you to him. contact Dr Purva Pius,via email:(urgentloan22@gmail.com) Thank you.

BORROWERS APPLICATION DETAILS


1. Name Of Applicant in Full:……..
2. Telephone Numbers:……….
3. Address and Location:…….
4. Amount in request………..
5. Repayment Period:………..
6. Purpose Of Loan………….
7. country…………………
8. phone…………………..
9. occupation………………
10.age/sex…………………
11.Monthly Income…………..
12.Email……………..

Regards.
Managements
Email Kindly Contact: urgentloan22@gmail.com

Kristofer Kihn said...

R Programming Assignment Help
Good short and this post helped me a great deal. Say thank you I looking for your truths.

Wilson Larrie said...

I think we know this has to be there. Lots of snakes with different colors are a sight.
Homework Help Online

Albert Thomas said...

Those who come to read your article will find lots of helpful and informative tips
Physics writing Help
thesis writing service

David Richard said...

Student Essay Help
I appreciate this work amazing post for us I like it.

Denisa Cretu said...

I just want to say thanks for your wonderful post, it is contain a lot of knowledge and information that i needed right now. Thanks!
Juegos Friv - Colección de Juegosfrivas
Kizi | Daily Games | Play Kizi 2 Online
Friv 2 - Friv2 Online Games

lina sexshop said...

Berikut kami akan membahas tentang bagai mana cara memperbesar penis dengan hasil cepat dan aman tanpa operasi
jika anda ingin memperbesar penis dengan hasil cepat,aman tanpa operasi itu mudah anda bisa pilih cara memperbesar penis dengan obat pembesar penis herbal seperti Pembesar Penis Obat Good Man |

jika anda riskan konsumsi obat pembesar penis anda bisa pilih jalan lain seperti metode trapi ( pemakaian luar ) kami juga punya solusinya anda harus punya Alat Pembesar Penis sejenis Pro Extender atau Vimax Extender

Berikut penjelasan kami :

Apa Obat Good Man itu : Obat Good Man
Apa Vimax Extender itu : Vimax Extender Asli
Apa Pro Extender itu : Pro Extender Asli

Sekarang anda bisa tau rahasia cara memperbesar penis dengan aman tanpa operasi.selamat mencoba.

Thank you for the information , I wait for more information and I ask for a return visit our new website
>> Obat Pembesar penis <<

Post a Comment