Markdown RAG Chunker API

Splits PDFs, web pages, Word, PowerPoint, Excel and EPUB files, or pasted Markdown, into header-aware chunks for RAG. Each chunk stays inside one section of the

Input

Send these Fields as input. A Field not listed here is refused.

markdown_file

Document URL (PDF, web page, Word, PowerPoint, Excel, EPUB, CSV, JSON, XML, Markdown or text)

  • Type: string
  • Required: no
  • Length in characters: at most 30000

document

Or upload the document (replaces Document URL)

An https URL or an upload id.

  • Type: string
  • Required: no

markdown_text

Or paste Markdown text

  • Type: string
  • Required: no
  • Length in characters: at most 30000

input_mode

Input mode: set it only when both a document and Markdown text are filled (file or text)

  • Type: string
  • Required: no
  • Allowed values: file, text

source_id

Source ID (optional stable ID for this document, e.g. the page URL of pasted text)

  • Type: string
  • Required: no
  • Length in characters: at most 2000

headers_to_split_on

Headers to split on

  • Type: list of strings
  • Required: no
  • Allowed values: #, ##, ###, ####, #####, ######

max_chunk_tokens

Max chunk size (tokens)

  • Type: number
  • Required: no
  • Value: 50 to 8192

tokenizer

Tokenizer (cl100k_base: OpenAI text-embedding-3; o200k_base: GPT-4o family)

  • Type: string
  • Required: no
  • Allowed values: cl100k_base, o200k_base

strip_headers

Move headers to metadata

  • Type: string
  • Required: no
  • Allowed values: Yes, No

max_chunk_chars

Max chunk size (characters): optional, a whole number from 100 to 100000; overrides the token size

  • Type: string
  • Required: no
  • Length in characters: at most 6

max_input_chars

Max document size (characters)

  • Type: number
  • Required: no
  • Value: 1 to 5000000

max_chunks

Max chunks per run

  • Type: number
  • Required: no
  • Value: 1 to 50000

Run it

One call presses the Widget and answers the run once its status is succeeded or failed, however long it takes. The key is read from ABYSS_API_KEY. Install the library with npm install abysshub or pip install abysshub; cURL sends the Fields as a form.

JavaScript

// index.mjs
import Abyss from "abysshub";

const abyss = new Abyss();

const run = await abyss.run("widget_384", {
  markdown_file: "https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md",
  headers_to_split_on: ["#", "##", "###"],
  max_chunk_tokens: 512,
  tokenizer: "cl100k_base",
  strip_headers: "Yes",
  max_input_chars: 2000000,
  max_chunks: 5000,
});

console.log(run.result);
for (const path of await run.save("output")) {
  console.log(path);
}

Python

from abysshub import Abyss

abyss = Abyss()

run = abyss.run("widget_384", {
    "markdown_file": "https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md",
    "headers_to_split_on": ["#", "##", "###"],
    "max_chunk_tokens": 512,
    "tokenizer": "cl100k_base",
    "strip_headers": "Yes",
    "max_input_chars": 2000000,
    "max_chunks": 5000,
})

print(run.result)
for path in run.save("output"):
    print(path)

cURL

curl https://api.abysshub.com/v1/widgets/widget_384/runs \
  -H "Authorization: Bearer $ABYSS_API_KEY" \
  -F markdown_file=https://raw.githubusercontent.com/yel-hadd/markdown-rag-chunker/main/README.md \
  -F 'headers_to_split_on[]=#' \
  -F 'headers_to_split_on[]=##' \
  -F 'headers_to_split_on[]=###' \
  -F max_chunk_tokens=512 \
  -F tokenizer=cl100k_base \
  -F strip_headers=Yes \
  -F max_input_chars=2000000 \
  -F max_chunks=5000

A succeeded run carries its result and its output_files. Their URLs are signed for each read: re-read the run, never store a URL. A run still queued or running in the answer (after an hour, or when the API is busy) is read again at its Location with ?wait=true.

What it answers

The Example Run's result, the JSON the Widget wrote to output/output.json:

{
  "input_mode": "file",
  "input_file_type": "text/plain; charset=utf-8",
  "input_chars": 27691,
  "chunk_count": 41,
  "elapsed_ms": 5404,
  "document_hash": "706ecf6eee932bcf",
  "unchanged": false,
  "charge_limit_reached": false
}

More

Keys, files, errors and limits: API access.