What Are Hugging Face Agents?. A step by step tutorial on how to use… | by Farzad Mahmoodinobar | May, 2023

6.2. Question Answering

Question answering is self-explanatory but let’s make it more interesting. Instead of providing a paragraph to the agent and asking questions about the provided information, let’s provide an image and ask the agent about the content of the image.

I wrote a few lines in a word document and then saved it as a *.jpg image in my local. Let’s first use PIL to see the image as follows:

from PIL import Image
image = Image.open('/content/jack.jpg')
image.show()

Results:

As you can see, the image has a few sentences about my imaginary friend in Seattle, named Jack. Next, let’s ask a question from our agent and see how it responds. I would like to ask the agent about Jack’s favorite color, as follows:

task = "in the following 'document', what is Jack's favorite color?"
agent_starcoder.run(task, document=image)

Results:

Once again, let’s look at the very last sentence in the bottom — That is pretty good! We can see that text extraction is not perfect, for example, it extracted Tesla to ‘tesia’ but still, the agent returned the relevant portion of the image, which answered our question.

Let’s see what tools exactly the agent used:

task = "in the following 'document', what is Jack's favorite color?"
agent_starcoder.run(task, document=image, return_code=True)

Results: