PDF Text Extractor

PDF Text Extractor allows you to extract text from PDF files. It also supports chunking of the text to prepare the data for usage with large language models.

Input

• URLs - URLs of the PDF files you want to extract the text from.
• Chunk size - the maximum size of a single chunk of text
• Chunk overlap - how many characters will overlap between neighbouring chunks of text

Output

Each item will contain the URL of the source PDF, index that identifies the position in the extracted text, and lastly, the extracted text.

Sample output

1[{
2  "url": "https://arxiv.org/pdf/2307.12856.pdf",
3  "index": 0,
4  "text": "Preprint\nA REAL-WORLD WEBAGENT WITH PLANNING,\nLONG CONTEXT UNDERSTANDING, AND\nPROGRAM SYNTHESIS\nIzzeddin Gur1∗ Hiroki Furuta1,2∗† Austin Huang1 Mustafa Safdari1 Yutaka Matsuo2\nDouglas Eck1 Aleksandra Faust1\n1Google DeepMind, 2The University of Tokyo\nizzeddin@google.com, furuta@weblab.t.u-tokyo.ac.jp\nABSTRACT\nPre-trained large language models (LLMs) have recently achieved better gener￾alization and sample efficiency in autonomous web automation. However, the\nperformance on real-world websites has still suffered from (1) open domainness,\n(2) limited context length, and (3) lack of inductive bias on HTML. We introduce\nWebAgent, an LLM-driven agent that learns from self-experience to complete tasks\non real websites following natural language instructions. WebAgent plans ahead by\ndecomposing instructions into canonical sub-instructions, summarizes long HTML\ndocuments into task-relevant snippets, and acts on websites via Python programs"
5},
6{
7  "url": "https://arxiv.org/pdf/2307.12856.pdf",
8  "index": 1,
9  "text": "generated from those. We design WebAgent with Flan-U-PaLM, for grounded code\ngeneration, and HTML-T5, new pre-trained LLMs for long HTML documents\nusing local and global attention mechanisms and a mixture of long-span denoising\nobjectives, for planning and summarization. We empirically demonstrate that our\nmodular recipe improves the success on real websites by over 50%, and that HTML￾T5 is the best model to solve various HTML understanding tasks; achieving 18.7%\nhigher success rate than the prior method on MiniWoB web automation benchmark,\nand SoTA performance on Mind2Web, an offline task planning evaluation.\n1 INTRODUCTION\nLarge language models (LLM) (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023) can\nsolve variety of natural language tasks, such as arithmetic, commonsense, logical reasoning, question\nanswering, text generation (Brown et al., 2020; Kojima et al., 2022; Wei et al., 2022), and even"
10},
11{
12  "url": "https://arxiv.org/pdf/2307.12856.pdf",
13  "index": 2,
14  "text": "interactive decision making tasks (Ahn et al., 2022; Yao et al., 2022b). Recently, LLMs have also\ndemonstrated success in autonomous web navigation, where the agents control computers or browse\nthe internet to satisfy the given natural language instructions through the sequence of computer\nactions, by leveraging the capability of HTML comprehension and multi-step reasoning (Furuta et al.,\n2023; Gur et al., 2022; Kim et al., 2023).\nHowever, web automation on real-world websites has still suffered from (1) the lack of pre-defined\naction space, (2) much longer HTML observations than simulators, and (3) the absence of domain\nknowledge for HTML in LLMs (Figure 1). Considering the open-ended real-world websites and the\ncomplexity of instructions, defining appropriate action space in advance is challenging. In addition,\nalthough several works have argued that recent LLMs with instruction-finetuning or reinforcement"
15}]

How to use PDF Text Extractor

Follow this tutorial to learn how to use PDF Text Extractor and combine it with LangChain to build an intelligent QA system that can extract answers from PDF documents.