c# - best use of Parallel.ForEach / Multithreading -

c# - best use of Parallel.ForEach / Multithreading -

i need scrape info website. have on 1,000 links need access, , dividing links 10 per thread, , start 100 threads each pulling 10. after few test cases, 100 threads best count minimize time retrieved content links.

i realized .net 4.0 offered improve back upwards multi-threading out of box, done based on how many cores have, in case not spawn plenty threads. guess asking is: best way optimize 1,000 link pulling. should using .foreach , allow parallel extension command amount threads spawned, or find way tell how many threads start , split work?

i have not worked parallel before maybe approach maybe wrong.

something worth checking out tpl dataflow library.

dataflow on msdn.

see nesting await in parallel foreach

the whole thought behind parallel.foreach() have set of threads , each processes part of collection. noticed, doesn't work async-await, want release thread duration of async call.

also, walkthrough creating dataflow pipeline sets , processes multiple web page downloads. tpl dataflow designed scenario.

c# .net multithreading parallel.foreach