Markdown RAG Chunker API
Splits PDFs, web pages, Word, PowerPoint, Excel and EPUB files, or pasted Markdown, into header-aware chunks for RAG. Each chunk stays inside one section of the
- Id:
widget_384 - Price: Ƀ 20 a run, plus the Call Price, Ƀ 0.1 an API run. Your first 100 API runs carry no Call Price.
- Widget page: Run it in your browser
- Docs: Call Abyss from your code
Input
Send these Fields as input. A Field not listed here is refused.
markdown_file
Document URL (PDF, web page, Word, PowerPoint, Excel, EPUB, CSV, JSON, XML, Markdown or text)
- Type: string
- Required: no
- Length in characters: at most 30000
document
Or upload the document (replaces Document URL)
An https URL or an upload id.
- Type: string
- Required: no
markdown_text
Or paste Markdown text
- Type: string
- Required: no
- Length in characters: at most 30000
input_mode
Input mode: set it only when both a document and Markdown text are filled (file or text)
- Type: string
- Required: no
- Allowed values:
file,text
source_id
Source ID (optional stable ID for this document, e.g. the page URL of pasted text)
- Type: string
- Required: no
- Length in characters: at most 2000
headers_to_split_on
Headers to split on
- Type: list of strings
- Required: no
- Allowed values:
#,##,###,####,#####,######
max_chunk_tokens
Max chunk size (tokens)
- Type: number
- Required: no
- Value: 50 to 8192
tokenizer
Tokenizer (cl100k_base: OpenAI text-embedding-3; o200k_base: GPT-4o family)
- Type: string
- Required: no
- Allowed values:
cl100k_base,o200k_base
strip_headers
Move headers to metadata
- Type: string
- Required: no
- Allowed values:
Yes,No
max_chunk_chars
Max chunk size (characters): optional, a whole number from 100 to 100000; overrides the token size
- Type: string
- Required: no
- Length in characters: at most 6
max_input_chars
Max document size (characters)
- Type: number
- Required: no
- Value: 1 to 5000000
max_chunks
Max chunks per run
- Type: number
- Required: no
- Value: 1 to 50000
Run it
One call presses the Widget and answers the run once its status is succeeded or failed, however long it takes. The key is read from ABYSS_API_KEY. Install the library with npm install abysshub or pip install abysshub; cURL sends the Fields as a form.
JavaScript
// index.mjs
import Abyss from "abysshub";
const abyss = new Abyss();
const run = await abyss.run("widget_384", {
markdown_file: "https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md",
headers_to_split_on: ["#", "##", "###"],
max_chunk_tokens: 512,
tokenizer: "cl100k_base",
strip_headers: "Yes",
max_input_chars: 2000000,
max_chunks: 5000,
});
console.log(run.result);
for (const path of await run.save("output")) {
console.log(path);
}Python
from abysshub import Abyss
abyss = Abyss()
run = abyss.run("widget_384", {
"markdown_file": "https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md",
"headers_to_split_on": ["#", "##", "###"],
"max_chunk_tokens": 512,
"tokenizer": "cl100k_base",
"strip_headers": "Yes",
"max_input_chars": 2000000,
"max_chunks": 5000,
})
print(run.result)
for path in run.save("output"):
print(path)cURL
curl https://api.abysshub.com/v1/widgets/widget_384/runs \
-H "Authorization: Bearer $ABYSS_API_KEY" \
-F markdown_file=https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md \
-F 'headers_to_split_on[]=#' \
-F 'headers_to_split_on[]=##' \
-F 'headers_to_split_on[]=###' \
-F max_chunk_tokens=512 \
-F tokenizer=cl100k_base \
-F strip_headers=Yes \
-F max_input_chars=2000000 \
-F max_chunks=5000A succeeded run carries its result and its output_files. Their URLs are signed for each read: re-read the run, never store a URL. A run still queued or running in the answer (after an hour, or when the API is busy) is read again at its Location with ?wait=true.
What it answers
The Example Run's result, the JSON the Widget wrote to output/output.json:
{
"input_mode": "file",
"input_file_type": "text/plain; charset=utf-8",
"input_chars": 27691,
"chunk_count": 41,
"elapsed_ms": 5404,
"document_hash": "706ecf6eee932bcf",
"unchanged": false,
"charge_limit_reached": false
}More
Keys, files, errors and limits: API access.