Skip to main content

Citations and Trace

In this AI4J chapter, citation and trace are easily lumped together as an "explainability capability." That statement is not wrong, but it is too coarse. In the source, these two actually solve two different problems:

  • citation answers "where the final context and answer can be traced back to"
  • trace answers "how hits change through the RAG pipeline"

This page covers the two chains separately.

1. Where the source entry points are​

There are few key classes, and they are highly concentrated:

  • rag/DefaultRagService.java
  • rag/DefaultRagContextAssembler.java
  • rag/RagContext.java
  • rag/RagCitation.java
  • rag/RagTrace.java
  • rag/RagResult.java

From this set of classes you can tell that AI4J's current citation/trace both belong to the RAG result assembly layer, not to a provider-layer capability.

2. How citations are generated​

What actually generates citations is neither the retriever nor the reranker, but rather:

DefaultRagContextAssembler.assemble(query, hits)

It processes each entry in the final hits order:

  1. Assigns a citationId to each hit, in the format S1, S2, S3
  2. Extracts from the hit:
    • sourceName
    • sourcePath
    • sourceUri
    • pageNumber
    • sectionTitle
    • content
  3. Generates a RagCitation
  4. Simultaneously assembles the context.text that goes to the model

So citations are not a separate table; they are produced together with the final context text handed to the model, inside the same assembler.

3. What RagCitation actually stores​

The current RagCitation is lightweight:

  • citationId
  • sourceName
  • sourcePath
  • sourceUri
  • pageNumber
  • sectionTitle
  • snippet

It does not include:

  • exact character offset
  • chunkId
  • dataset
  • hit score

This shows that AI4J's current citation design targets lean toward:

  • giving the model and the user a readable source label
  • providing basic source hints for troubleshooting

rather than fine-grained, audit-grade citation localization.

4. How the citation prefix in the context text is assembled​

By default, DefaultRagContextAssembler prepends something like the following to each hit's content:

[S1] source label

The source label is assembled with roughly this priority:

  1. sourceName
  2. sourcePath
  3. otherwise falls back to source

If present:

  • pageNumber
  • sectionTitle

they are also appended to the label.

Only then is hit.content appended.

This shows that in AI4J's current implementation, citations do not merely return structured fields; they are written directly into the context text the model sees.

5. What includeCitations actually controls​

RagQuery.includeCitations defaults to true.

But looking at the DefaultRagContextAssembler implementation, what it controls is:

  • whether to write prefixes like [S1] source label into context.text

It does not control whether the RagCitation list is generated.

That is, even if you set includeCitations = false:

  • RagResult.citations will still have content
  • only the context handed to the model no longer embeds citation labels

This detail is easily misread as "turning it off means no citations", which is not actually the case.

5.1 What to do when you need to control the context token budget​

By default, DefaultRagContextAssembler does no token truncation; it only assembles the context in the final hits order. If your RAG result goes directly into the model prompt, you should explicitly swap in TokenAwareRagContextAssembler:

RagService ragService = new DefaultRagService(
        retriever,
        reranker,
        new TokenAwareRagContextAssembler("gpt-4o-mini", 3000)
);

TokenAwareRagContextAssembler's token count is a context budget guard, not a precise metering tool. It is recommended to pass the model name you actually use first; if the underlying tokenizer does not yet recognize the model name, it will automatically fall back to the default cl100k_base estimate, so RAG does not fail just because a new model name cannot be parsed.

If you know exactly which tokenizer the model uses, you can also explicitly override the encoding:

RagContextAssembler assembler = TokenAwareRagContextAssembler.withEncoding(
        EncodingType.O200K_BASE,
        3000
);

If both the model name and encoding are uncertain, just use new TokenAwareRagContextAssembler(3000) and set the budget conservatively.

It does only three things:

  • adds context in the existing hit order until the token budget is reached;
  • when the first hit alone is too long, truncates that hit's content;
  • RagCitation only returns the sources that actually made it into the context.

Without configuration it still goes through DefaultRagContextAssembler, leaving default behavior unchanged.

6. How trace is generated​

trace is produced by DefaultRagService.search(...).

Only when:

query != null && query.isIncludeTrace()

will the result carry:

RagTrace.builder()
    .retrievedHits(hits)
    .rerankedHits(reranked)
    .build()

That is, DefaultRagService.search(...) only records the retrieval chain by default:

  • post-retrieval hit list
  • post-rerank hit list

It is not responsible for recording:

  • the final model answer
  • prompt assembly details
  • provider reasoning

RagTrace reserves a generationUsage field. After the upper-layer ask/plugin/demo calls the model to generate an answer, it can backfill the response usage and the cost computed against the business price table; core RAG does not ship a price table, nor does it kick off final answer generation inside search(...).

So do not misrepresent it as "automatic full-chain trace".

7. How to wire in the online LLM judge​

RagService.search(...) only does retrieval and context assembly; it does not generate the final answer. So online evaluation does not run automatically inside search(...); instead, you call it explicitly after you have the final answer:

RagResult rag = ragService.search(RagQuery.builder()
        .query("What is PTO?")
        .includeTrace(true)
        .build());

String answer = chatWithContext(rag.getContext());

RagOnlineEvaluator evaluator = aiService.getRagOnlineEvaluator(
        PlatformType.OPENAI,
        "gpt-4o-mini"
);

RagJudgeEvaluation judge = evaluator.evaluate(rag, answer);

Double faithfulness = judge.getFaithfulnessScore();
Double contextRelevance = judge.getContextRelevanceScore();
Double answerRelevance = judge.getAnswerRelevanceScore();

The built-in ChatRagJudge does one thing: it sends the question / answer / retrieved context to a chat model and asks it to return a JSON score. The result is written back into:

rag.getTrace().getJudgeEvaluation()

If you do not want to use the built-in prompt or chat provider, implement it directly:

class MyJudge implements RagJudge {
    public RagJudgeEvaluation judge(RagJudgeRequest request) {
        // call your evaluator / policy / judge model
    }
}

This is not a replacement for offline Recall/MRR. It is better suited for online sampling, debugging, and quality replay. The three scores commonly checked are:

  • faithfulnessScore: whether the answer is faithful to the retrieved context
  • contextRelevanceScore: whether the recalled context is relevant to the question
  • answerRelevanceScore: whether the answer actually addresses the question

8. Why trace cannot be replaced by the final answer alone​

The final answer only tells you the model output. But the questions that really come up in RAG troubleshooting are:

  • Did recall miss entirely?
  • Did recall hit but the ranking is wrong?
  • Is the ranking right but the context is too long and diluted by noise?

Although the current RagTrace is lightweight, it at least helps you distinguish the first two:

  • retrievedHits to inspect the recall set
  • rerankedHits to inspect ranking changes

This matters a great deal for tuning:

  • dense topK
  • hybrid fusion
  • rerank topN
  • finalTopK

9. Why citation and trace draw from different sources​

These two draw from different pipeline stages.

citation comes from:

  • the finalHits eventually passed to contextAssembler

trace comes from:

  • retrievedHits
  • rerankedHits

This means citation is a post-state, while trace is closer to an intermediate-state record.

A direct consequence:

  • a hit trimmed by finalTopK may still appear in trace
  • but it will not appear in citations

So if you see a hit that is "in trace but not in citations", it is not necessarily a bug; it most likely just did not make it into the final context.

10. What actually affects citation quality​

From the source, a citation's content is taken almost directly from the RagHit. So citation quality is affected first by the upstream layers:

  • whether chunking is reasonable
  • whether metadata is complete
  • whether the retriever brings back sourceName/sourcePath/pageNumber/sectionTitle
  • whether content was replaced after rerank

DenseRetriever in particular tries to recover from metadata:

  • documentId
  • sourceName
  • sourcePath
  • sourceUri
  • pageNumber
  • sectionTitle
  • chunkIndex

If these fields were not stored during ingestion, citations can only ever be weak no matter how they are written.

11. The truest boundaries of the current design​

AI4J's citation/trace layer is useful, but the boundaries need to be spelled out.

It currently does not directly provide:

  • chunk-level exact offset localization
  • automatic binding of spans in the final answer to citations
  • prompt version history
  • causal proof of which citation the model used
  • provider-level traceability

The LLM judge can give online quality an observable score, but it is itself a model judgment, not strong proof.

So it is closer to:

  • a readable citation mechanism
  • a lightweight RAG process snapshot

than a legal-grade, audit-grade, or research-grade citation system.

12. The 6 most common pitfalls​

12.1 Assuming citations come from the original document​

Citations actually come from the final RagHit, not directly from the original document object.

12.2 Assuming turning off includeCitations removes the citation structure​

Turning it off only removes the label prefix in the context text, not the RagCitation list itself.

12.3 Assuming trace is a full call chain​

Trace currently covers retrieval and rerank by default; generationUsage only appears if the upper layer backfills usage/cost after generating the answer; judge scores are only written if you explicitly call RagOnlineEvaluator.

12.4 Ignoring metadata quality​

For hits without sourceName, pageNumber, or sectionTitle, citation readability drops noticeably.

12.5 Equating citation with answer grounding​

AI4J can provide citation material; that does not mean the model necessarily answers strictly according to those citations.

12.6 Treating LLM judge scores as strong facts​

warning

The LLM judge is an online quality signal, not audit proof. High-risk scenarios should still keep manual sampling or rule-based validation.

13. Where the safest extension points are​

If you want to enhance citation/trace, the current safest extension points are:

  • a custom RagContextAssembler
  • improving ingest metadata
  • backfilling generationUsage from the upper-layer runtime after the answer is generated
  • calling RagOnlineEvaluator after answer generation

Do not push everything onto the retriever. The retriever is responsible for fetching chunks, not for deciding the final citation presentation format.

14. The conclusion worth remembering from this page​

AI4J's current citation and trace are not the same thing:

  • citations are generated by DefaultRagContextAssembler based on the final hits
  • trace is recorded by DefaultRagService across the retrieval / rerank stages
  • generation usage is backfilled into trace explicitly by the upper layer after generating the answer
  • judge evaluation is written into trace explicitly by RagOnlineEvaluator after the final answer

The former targets "final citation and context presentation"; the latter targets "intermediate-process troubleshooting". Keeping these two layers distinct keeps RAG explainability analysis from getting muddled.

15. Further reading​