Cooking with NLP
Web Scraping recipes and analyzing scraped data
A little context on “why this project?”
A few months ago, I moved to the US from India. Back home, I never worried about groceries. Everything was always taken care of! (perks of living with family I suppose) However, this changed after I moved here. I always thought about how I needed to stop for milk on my way back, or how I needed to order rice so that it arrives before we run out of it. This was something anyone who lives by themself goes through, right? But sometimes, I used to realize that I was out of something right when I need it to cook a dish that I was so excited to cook and eat! This annoyed me. I am sure people who are extremely organized don’t have experiences like these. But other people who relate to me, I have decided to do something about this problem!
After multiple such occurrences, the Data Scientist in me was furious, and decided to solve this problem using Data!
What I want at the end of the project is to exactly know what I didn’t have at home to make a dish that I want (without looking through the recipe and checking every ingredient at home). What dishes I could make with the ingredients at home? And by extension, learn more about the food that I love!
By now you might have realized, I take food way too seriously XD.
Truer words have never been spoken!
The problem that we discussed above is a broad problem, and the approach to solve it directly makes no sense! What I plan to do is break it into smaller, achievable problems. Divide and conquer!
So, for the first part of this project, I try to approach a rather simple problem: Find out the most common ingredients in a set of recipes!
Let’s dissect the problem further.
Problem Statement:
Scrape at least 100 recipes from the web, provide their ingredient lists and clean the ingredient data for further calculation.
Tasks:
- Scraping data: scrape at least 100 recipes from any recipe website.
- Cleaning data: clean the ingredient data for further calculation. (This includes, but is not limited to, removing excess white spaces, correcting for all edge cases, and correcting any remaining formatting issues)
- Calculating: what are the 10 most common ingredients used in these recipes?
While starting this problem, I came across this website called ‘allrecipes’. While analyzing the structure of the website to begin scrapping, I realized that the website is very well structured and it would be fairly easy to extract data from this. Also, the website has a lot of recipes from various cuisines and seems to be well maintained and moderated in terms of the recipes. So, I chose to work with this website.
Let’s get on to it then!
Task 1: Web Scrapping Recipes
There were two approaches that I considered to complete this task.
- Approach 1: To extract the required information from the HTML front end, directly use code to download the HTML contents and extract useful information using requests and regex.
- Approach 2: To extract the required information from the HTML front end, directly use code to download the HTML contents and extract useful information using requests and Beautiful Soup.
About:
- Regular Expression (shortened as regex): It is a sequence of characters that specifies a search pattern in the text.
- Requests: It is a Python module in which you can send HTTP requests to retrieve contents. It helps you to access website HTML contents by sending Get or Post requests.
- Beautiful Soup: It helps you parse the HTML or XML documents into a readable format. It allows you to search different elements within the documents and help you retrieve required information faster.
Step 1: Understanding the HTML
After surfing the website for a couple of minutes, I understood that there is a base URL to all the recipes on the website: https://www.allrecipes.com/recipe/ So, I checked out if all of these recipes with the base URL have a similar HTML structure. To do this, I took a couple of pages at random and checked the structure. For demonstration here, I have provided the HTML structure of one such page. You can do it using the following code:
This is what the HTML structure looks like:
Check out the entire HTML structure by simply running the code above.
When we look at the entire HTML structure, we see that there is a lot of unwanted information. But, we understand a few key points from this which makes our task fairly simpler.
- The title of the HTML page gives us the name of the recipe with some minor modifications. Hence, regex can be used to extract the name of the recipe.
- The ingredients of that particular recipe are stored as a list with the key “recipeIngredient” for all the pages. Hence, regex with some simple functions can give us a list of ingredients.
- Even though the URL to any recipe is our input, in this case, we can extract the complete URL from the HTML using the key “url”
Using this information, we can carry out the next step.
Step 2: Scrape the website allrecipes.com
Step 3: Combining the data scraped into a data frame
Task 2: Cleaning Scraped Data
On investigating the 109 scraped recipes, I observed that there are several formatting “edge cases” unique to this website. These edge cases were not at all related to the ingredient names. These edge cases are:
- Measurements are represented as ½ (called vulgar fractions)
- Several non-alphanumeric characters such as copyright and trademark symbols are used to identify ingredients, comma as used in “½ onion, finely chopped”, brackets as used in “1 (1 ounce) envelope dry onion soup mix”, hyphens (-) as used in “all-purpose flour”
I will clean the data in two phases:
- Primary Cleaning: The objective of the first phase is to ensure that the data is readable and accessible on all platforms by fixing encoding errors and eliminating symbols that aren’t translated well across platforms. This cleaning will not get rid of any punctuations, stopwords, etc.
- Problem-specific cleaning: The objective of the second phase of the cleaning is to prepare the data for our calculations and is centered on the problem set requirement.
I believe it is a good practice to separate the both as if requirements change in the future you can always proceed with the result of the first cleaning phase to perform another analysis altogether.
Step 1: Primary Cleaning
Step 2: Problem-Specific Cleaning
The objective is to extract the ingredient name from sentences that contain additional information such as measurement, unit of measurement, and ingredient state-specific information (chopped, minced, frozen, etc).
Data Exploration
In order to eliminate the additional information, the position of the additional information w.r.t the ingredient name is helpful. It is indicative of the position of the ingredient and almost follows a pattern though not strictly.
A few patterns and their example are:
- Pattern: quantity measurement ingredient Example: 1 teaspoon soy sauce
- Pattern: quantity ingredient Example: 2 eggs
- Pattern: quantity ingredient, ingredient-specific information Example: 1⁄2 onion, finely chopped
A few other patterns can be observed here
Let’s see if there are any overlaps in this cleaned data.
Ingredient Extraction Methodology via Named Entity Recognition
Since there is a dependency among the components of the sentence and we know that the ingredient name will be a noun, we can use this information along with custom regex expressions to eliminate measurement units to extract the ingredient name.
Here, to find the ingredient,
PSEUDOCODE
for each token do the following:
- If on checking the token dependencies, the dependencies of the token for sentences’ subject or root are true then move to step 2.
- If the token is a noun, then move to step 3.
- Scan the token for children which are either modifications or compounds and not measurements and return the identified token as the ingredient name.
Task 3: Analysis and Calculation
Step 1: Count Calculation
Step 2: Proportion Calculation
Let us find if one ingredient appears more than once in a recipe. This is important as if they don’t appear more than once then the count divided by the number of recipes will give us the proportion.
However, if an ingredient occurs more than once the count is not reflective of the number of recipes it occurs in alone and includes multiple occurrences within a recipe.
Since there are a few recipes that contain the same ingredient multiple times. This could be because there could be variations of the ingredient such as chopped, diced onions, etc.
Thus, I first group the recipes by name and find the set of ingredients associated with each and then count each ingredient's occurrence to eventually calculate the proportion.
The top 10 ingredients are filled with condiments and dairy mainly. The only vegetable here is garlic since it is used in almost all sauces, gravies, etc. There is flour too and the presence of this along with eggs and butter suggests substantial baking recipes among the scraped dataset.
Funnily enough, water is the top 5th ingredient, even though it is used in almost all recipes. This is because some of the recipes don’t consider water as an ingredient.
We see that the 10th ingredient is ‘package’ and it was not properly removed during the cleaning step.
