Ingest and retrieve documents

Outcome: add a document to the shared knowledge collection and see it returned as a source. Use a disposable local lab with sample content.

Ingest one document

After setting your API key, run in WSL Bash:

curl --fail-with-body http://localhost:8000/api/v1/ingest \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '[{"id":"academy-example-001","content":"The fictional Aurora service uses port 8123 in the training lab."}]'

Expected result: status is success and ingested is 1. Use a new ID for new content; this endpoint calls add, not a documented update/upsert operation. Avoid treating a repeated success response as proof an existing document changed.

Retrieve the source

curl --fail-with-body http://localhost:8000/api/v1/query \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"query":"Which port does the fictional Aurora service use?","context_limit":1,"include_sources":true,"user_id":"lab-user"}'

Inspect both the answer and sources. The stored sentence should be visible in retrieved content. A correct-looking answer without the source is insufficient evidence that retrieval worked.

Build a corpus from repository examples

The repository includes backend/scripts/extract_devsecops_knowledge.py. Run it only in a development copy containing material you intend to ingest; it recursively inspects Ansible YAML, Terraform files, and GitLab CI configuration.

.venv/bin/python backend/scripts/extract_devsecops_knowledge.py

It writes data/devsecops_corpus.json. Inspect the output before ingestion: its Terraform parsing is basic, and some CI structures can be skipped after parser errors. A file being created does not prove all intended sources were extracted.

The output wraps documents in a top-level object, while the API expects a JSON array. Convert only the document ID and content fields into the API shape:

.venv/bin/python - <<'PY'
import json
from pathlib import Path
corpus = json.loads(Path("data/devsecops_corpus.json").read_text())
documents = [{"id": item["id"], "content": item["content"]}
             for item in corpus["documents"]]
if not documents:
    raise SystemExit("No documents extracted; inspect source files and extraction logs.")
Path("data/ingest-documents.json").write_text(json.dumps(documents))
print(f"Prepared {len(documents)} documents; review the file before ingestion.")
PY

After reviewing the file and setting the lab API key:

curl --fail-with-body http://localhost:8000/api/v1/ingest \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  --data-binary @data/ingest-documents.json

Confirm the count and query for a known fact from one document. Extraction IDs can repeat between runs; plan stable unique IDs before combining corpora. The current API does not preserve extraction metadata or update existing IDs reliably.

Limits to understand

• The API stores the supplied document as one embedding; chunk long documents before ingestion.
• Custom metadata is not preserved by this endpoint; it records the document ID and timestamp.
• Knowledge is shared across callers. It is not a per-user access-controlled index.
• The backend can substitute a zero embedding on embedding failure. Check logs if retrieval quality is unexpected.
• There is no public knowledge-document deletion endpoint. Keep this exercise in a disposable store; do not delete a live vector directory to clean up a lab.

Next: Complete the RAG lab or understand the data flow.