
Resources
Consolidating and Exploring Open Textual Knowledge Prof. Ido Dagan, Bar Ilan University >> Click here
מבוא לשפה - עיבוד ממוחשב של שפה אנושית עם פרופסור עידו דגן With Spotify
Start with NLP
Recommended courses:
https://www.coursera.org/specializations/natural-language-processing
Recommended textbook, available online:
https://web.stanford.edu/~jurafsky/slp3/
It also provides great little introductions to many fields of linguistics before you hop into the computational part.
NLP Tutorials Part -I from Basics to Advance
https://www.analyticsvidhya.com/blog/2022/01/nlp-tutorials-part-i-from-basics-to-advance/
Natural Language Processing with Python
100 ChatGPT terms explained from NLP to Entity Extraction
https://www.geeky-gadgets.com/chatgpt-terms-explained/
Natural Language Processing In Healthcare
Natural Language Processing Specialization
https://www.coursera.org/specializations/natural-language-processing#courses
Hebrew NLP Resources
https://github.com/NNLP-IL/Resources
NNLP-IL Hebrew and Arabic NLP Resources
https://resources.nnlp-il.mafat.ai
מאגרי מידע ושת"פים אפשריים
https://docs.google.com/spreadsheets/d/1fGYKyA5Jf_KPCXPCpRWGfRzjDc6ALp9dgKnbIXqxM_Y/edit#gid=0
חוות דעת: שימושים בתכנים מוגנים בזכויות יוצרים לצורך למידת מכונה
https://www.gov.il/he/departments/legalInfo/machine-learning
Open Source
Github
NLP
https://github.com/topics/natural-language-processing
Speech
https://github.com/topics/speech
spaCy · Industrial-strength Natural Language Processing in Python
https://spacy.io/
Stanza – A Python NLP Package for Many Human Languages
Created by the Stanford NLP Group
https://stanfordnlp.github.io/stanza/a
Open Source OCR
https://github.com/tesseract-ocr/tesseract
Speech Recognition - Whisper (OpenAI)
https://cdn.openai.com/papers/whisper.pdf
Large language model (LLM)
Open LLMs List
Meta Llama 2
https://ai.meta.com/llama/
https://github.com/eugeneyan/open-llms
What’s before GPT-4? A deep dive into ChatGPT
https://medium.com/digital-sense-ai/whats-before-gpt-4-a-deep-dive-into-chatgpt-dfce9db49956
GPT-4 Training process
Like previous GPT models, the GPT-4 base model was trained to predict the next word in a document, and was trained using publicly available data (such as internet data) as well as data we’ve licensed. The data is a web-scale corpus of data including correct and incorrect solutions to math problems, weak and strong reasoning, self-contradictory and consistent statements, and representing a great variety of ideologies and ideas.
So when prompted with a question, the base model can respond in a wide variety of ways that might be far from a user’s intent. To align it with the user’s intent within guardrails, we fine-tune the model’s behavior using reinforcement learning with human feedback (RLHF).
Note that the model’s capabilities seem to come primarily from the pre-training process—RLHF does not improve exam performance (without active effort, it actually degrades it). But steering of the model comes from the post-training process—the base model requires prompt engineering to even know that it should answer the questions.
https://openai.com/research/gpt-4
BERT
https://github.com/google-research/bert
AlephBERT
https://github.com/OnlpLab/AlephBERT
https://arxiv.org/pdf/2104.04052.pdf
Multi-language Aspects
How Language-Neutral is Multilingual BERT?
https://arxiv.org/pdf/1911.03310.pdf
AraBERT: Transformer-based Model for Arabic Language Understanding
https://arxiv.org/pdf/2003.00104.pdf
ELMo
https://allennlp.org/elmo
LaBSE - Language-agnostic BERT sentence embedding model supporting 109 languages.
https://tfhub.dev/google/LaBSE/2
LaBSE model to PyTorch. It can be used to map 109 languages to a shared vector space.
https://huggingface.co/sentence-transformers/LaBSE
Claude is a large language model (LLM) built by Anthropic.
It's trained to be a helpful assistant in a conversational tone.
https://docs.anthropic.com/claude/docs/getting-started-with-claude
Jais - open-source Arabic Large Language Model (LLM)
https://huggingface.co/core42/jais-13b/tree/main
https://inceptioniai.org/jais/