What is the idea? The problem we want to solve
Nowadays companies receive hundreds of application for a single job, human resources and management find themselves screening a massive amount of resumes to find the right candidates for a single position. What if AI could help us make this process faster and more accurate? At Innoprox Labs we decided to answer this question with the help of NLP and LLMs like Llama.
The challenge is to provide a solution that compares a given job position with all documents sent by the applicants and rank the resumes from highest to lowest match based on their similarity. In the field of NLP this is a text classification and semantic similarity task. As a solution we propose the use of NLTK to pre-process all documents, a LLM to generate embeddings for each resume and the job description and a similarity algorithm to compare every resume embeddings with the job description embeddings.
The solution we propose
For the data pre-processing we used NLTK. This toolkit comes with a suite of text processing libraries that facilitates tasks like tagging, tokenization and the removal of stop words that are not relevant for semantic analysis. When choosing between word and sentence tokenization is important to understand the impact of each strategy:
- Tokenizing words: we can compare texts by examining the relationships between individual words. This type of tokenization can capture subtle differences in word choice and paraphrasing.
- Tokenizing sentences: it can capture the overall meanings between sentences, which is more robust to variations in word choice. It focuses more in the overall message than the word usage, which can be useful to identify different ideas expressed in different ways.
Considering that resumes can be one to several pages long and job skills and experience can be described in multiple ways it makes more sense to choose sentence tokenization. This also requires less computational resources due to a smaller feature space. Nevertheless, we decided to try both and our guess was right! Sentence tokenization provides an overall better accuracy for this task. Here is an example of how to use NLTK for these tasks:
fom nltk import pos_tag, sent_tokenize
from nltk.corpues import stopwords
resume_features = []
stop_words = set(stopwords.words("english"))
sentences = sent_tokenize(text)
for sentence in sentences:
words = [word for word in sentence if word not in stop_words]
tagged_words = pos_tag(words)
filtered_words = [word for word, tag in tagged_words if tag not in ['DT', 'IN', 'TO', 'PRP', 'WP']]
resume_features.extend(filtered_words)For generating the embeddings we chose to use 2 pre-trained models: BERT and Llama. BERT is a model that uses a transformer encoder architecture and is widely used in tasks like text classification, sentiment analysis and sentence similarity. Llama is a transformed based language model that focuses on efficient scaling and development, it is efficient for a wide range of tasks, including text generation and understanding, conversational AI and more advanced NLP tasks. It is also a much newer model that have been upgraded in the last years with the last version being Llama 3 being released on 2023.
Our choice of using BERT is to provide a way to run our solution in any laptop, specially since Llama requires a large amount of RAM and a high end GPU to generate embeddings in a reasonable time. This also allowed us to perform hundreds of test on a small model first to evaluate the impact of text pre-processing and features generation. To facilitate the use of multiple models and the option to add more in the future we used Hugging Face, that is the transformers package. It provides easy access to several pre-trained and fine-tunned models after creating an account, requesting access and generating an access token. Here is an example of how to use the transformers library with Llama 2:
from transformers import AutoModel, AutoTokenizer, AutoModelForTokenClassification, AutoModelForCausalLM
import torch
access_token = "your-hf-token"
model_name = "meta-llama/Llama-2-7b-hf
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(model_name, token=access_token)
tokenizer = AutoTokenizer.from_pretrained(model_name, token=access_token)
tokenizer.pad_token = tokenizer.eos_token"
model.to(device)
resume_embeddings = []
batch_size = 16
with torch.no_grad():
for i in range(0, len(resume_features), batch_size):
batch_features = features[i:i+batch_size]
inputs = tokenizer(batch_features, return_tensors="pt", truncation=True, padding=True, max_length=512).to(device)
outputs = model(**inputs, output_hidden_states=True)
hidden_states = outputs.hidden_states[-1]
embeddings = hidden_states.mean(dim=1).detach().to(device).numpy()
resume_embeddings.extend(embeddings)For the job and resumes embeddings comparison task we chose as metric cosine similarity. This metric is widely used to measure similarity between sentences, paragraphs or documents, where each document is represented as a vector of words frequencies. Cosine similarity measures how similar 2 vectors/embeddings are and it is easy to compute and understand , which makes it suitable for our task.
For every resume a similarity matrix is generated by comparing the similarity of all the embeddings from the job description with all the embeddings from the resume. The mean of this matrix gives us the similarity score or the ranking of the resume in question. With all the scores we generate a table that lists all resumes and their score ranked from highest to lowest based on their similarity with the job description. The cosine similarity can be calculated using SK-Learn:
from sklearn.metrics.pairwise import cosine_similarity
def get_resume_score(resume_embeddings):
similarity_matrix = cosine_similarity(job_embeddings, resume_embeddings)
return np.mean(similarity_matrix)We discovered that for this specific task the matching process can be further enhanced by finding entities in the job description. In this case entities can represent important pieces of information like skills, job titles, qualifications and other relevant keywords for the job. This task is known as Named Entity Recognition (NER) and for this propose we used a version of BERT and Llama that is fined-tuned for entities extraction. This information produces more precise and relevant embeddings that capture the core requirements of the job, leading to more accurate matching with resumes that contain similar entities. This can be also used to filter out resumes that do not match key criteria making the selection process more efficient. We added this enhancement as optional in the matching process in order to allow the user to decide if this feature increases the matching accuracy or not depending on how specific are the skills or experience required for the role.
In order to give a better score to the resumes that contain this keywords we added weights to the embeddings, where the ones derived from the extracted entities have a higher weight than the general embeddings derived from the rest of the next.
weights = {
'skills': 0.4,
'key_skills': 0.6
}
embeddings = resume_embeddings * weights['skills']
key_embeddings = key_resume_embeddings * .weights['key_skills']Legal and practical impacts
It is important to consider that for this solution we didn’t train or fine-tuned any mode, that is the case because we don’t have access to data outside our own or publicly available data sets, with AI generated or synthetic data. We extracted text from these datasets to generate embeddings and perform comparisons. The reason for this is that we are unclear about legal implications and workers when a machine is pre-selecting the best candidate for a job, and we want to be compliant with GDPR. All the packages and licenses we have used are open-source or under specific licenses that allow free use under certain terms & conditions. Therefore take into consideration that Llama has license limitations. Additionally we don’t give legal advice, this is a lab projects we developed at Innoprox to understand what needs to be look upon if such a solution would be viable and usable in a company or enterprise.
Our conclusions and lessons learned about this lab project at Innoprox
As expected the choice of the LLM does make an impact on the accuracy and matching process. Llama for example is a much larger model, trained in diverse and more extensive dataset, thus it has 7B to 70B parameters in comparison to BERT that only has 110M. Models like Llama are computational expensive to run but they have a better understanding of language, leading to better contextual embeddings, which are crucial for tasks requiring deep understanding of text or in this case resumes. A clear limitation of BERT is the maximum token limit at 512 which can impose a problem when processing long texts like several pages resumes or job description. This requires to divide the text in chunks at the expense of losing semantic understanding. Models like Llama don’t have this limitation and can handle longer documents to capture long range dependencies. Nevertheless we got acceptable results from BERT, in a dataset of 100 resumes the model was able to place most of the best suited candidates among the top of the list. For best results and further improvement of this project fine-tuning on a specific dataset of resumes, job descriptions could be done.