Introduction
Welcome to my 98th blog post on the Python journey. On day 98, I learned about Multiprocessing in Python which is used to run multiple processes in parallel. Let's dive into more details and understand the basics and functions of multiprocessing in Python.
So let's get started......
Multiprocessing in Python
It provides a simple way to run multiple processes in parallel.
Using multiprocessing we can take advantage of multiple cores or processors on our system and we can significantly improve the performance of our code.
Importing Multiprocessing
Syntax - We use multiprocessing by importing a multiprocessing module.
import multiprocessing
Creating a process
Similar to multi-threading, in multiprocessing we need to create a process object which calls a start() method. This start() method runs the process and then to stop the execution, we use the join() method.
Syntax to create a simple process
import multiprocessing
def my_func():
print("Hello from process", multiprocessing.current_process().name)
process = multiprocessing.Process(target=my_func)
process.start()
process.join()
Multiprocessing Functions
Below are the most commonly used Multiprocessing Functions in Python
multiprocessing.Process(target,args): It creates a new process that runs the target function with the specified arguments.
multiprocessing.Pool(processes): creates a pool of worker processes that can be used to parallelize the execution of a function across multiple input values.
multiprocessing.Queue():creates a queue that can be used to communicate data between processes.
multiprocessing.Lock():creates a lock that can be used to synchronize access to shared resources between processes.
Creating a pool of worker processes
Here the idea is to create a pool of worker processes and then assign tasks to them as and when needed. Pooling helps us take advantage of multiple CPU cores and process tasks in parallel, thus enabling the task to be performed efficiently and quickly.
Example -
from multiprocessing import Pool
def process_task(task):
# Do some work here
print("Task processed:", task)
if __name__ == '__main__':
tasks = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
with Pool(processes=4) as pool:
results = pool.map(process_task, tasks)
Using a queue to communicate between processes
While working with multiple processes, we can pass data using a queue. This allows the data to be added at one end and removed at the other end. In the context of multiprocessing, a queue can be used to pass data between processes.
Example -
def producer(queue):
for i in range(10):
queue.put(i)
def consumer(queue):
while True:
item = queue.get()
print(item)
queue = multiprocessing.Queue()
p1 = multiprocessing.Process(target=producer, args=(queue,))
p2 = multiprocessing.Process(target=consumer, args=(queue,))
p1.start()
p2.start()
Using a lock to synchronize access to shared resources
Similar to multi-threading, in multiprocessing, locks can be used to synchronize access to shared resources among multiple processes.
A lock is an object that acts as a semaphore, allowing only one process at a time to execute a critical section of code.
The lock is released when the process finishes executing the critical section.
def increment(counter, lock):
for i in range(10000):
lock.acquire()
counter.value += 1
lock.release()
if __name__ == '__main__':
counter = multiprocessing.Value('i', 0)
lock = multiprocessing.Lock()
p1 = multiprocessing.Process(target=increment, args=(counter, lock))
p2 = multiprocessing.Process(target=increment, args=(counter, lock))
p1.start()
p2.start()
p1.join()
p2.join()
print("Counter value:", counter.value)
Example -
import concurrent.futures
import requests
def downloadFile(url, name):
print(f"Started Downloading {name}")
response = requests.get(url)
open(f"files/file{name}.jpg", "wb").write(response.content)
print(f"Finished Downloading {name}")
url = "https://picsum.photos/2000/3000"
# pros = []
# for i in range(50):
# # downloadFile(url, i)
# p = multiprocessing.Process(target=downloadFile, args=[url, i])
# p.start()
# pros.append(p)
# for p in pros:
# p.join()
with concurrent.futures.ProcessPoolExecutor() as executor:
l1 = [url for i in range(60)]
l2 = [i for i in range(60)]
results = executor.map(downloadFile, l1, l2)
for r in results:
print(r)
Resources Used
You can watch the video of Day#98 by clicking on the below link 👇👇👇👇👇
Conclusion
Thanks, guys for going through this blog post. On day 98, I learned about multiprocessing in Python which is a special module that provides a simple and efficient way to run multiple processes in parallel. The multiprocessing module can be used to create a new process, run a function across multiple input values, communicate data between processes, or synchronize access to shared resources etc.
Thank you if you read this post and have found this post useful. I hope you have joined me and are enjoying my magical journey of Python coding. This is it for Day #98
See you in the next one.....
About Me
Hey Guys, I am Chintan Jain from CodeWithJain. I am a trader and content creator. I am also passionate about tech and hence wanted to explore the field of tech. I always wanted to learn to code so I watched many tutorials but procrastinated practicing coding. To get into the habit of coding consistently I am starting to BLOG with HASHNODE daily.
I will document my coding journey from scratch and share my daily learnings in a blog post on HASHNODE. I hope you all will enjoy my content and my coding journey.
So what are you waiting for, smash the FOLLOW and LIKE buttons and follow along my coding journey, a step to create more beautiful digital products and empower people.