当前位置：首页 > news >正文

langchain4j笔记-09

news 2026/5/28 14:26:26

RAG

1. easy rag

@Testvoidtest03(){// 1. 创建模型// 2. 加载文档List<Document>documents=ClassPathDocumentLoader.loadDocuments("excel");//List<Document> documents = FileSystemDocumentLoader.loadDocuments("/home/langchain4j/documentation");// 递归去加载// List<Document> documents = FileSystemDocumentLoader.loadDocumentsRecursively("/home/langchain4j/documentation");// 通过正则表达式去过滤对应的文件//PathMatcher pathMatcher = FileSystems.getDefault().getPathMatcher("glob:*.pdf");//List<Document> documents = FileSystemDocumentLoader.loadDocuments("/home/langchain4j/documentation", pathMatcher);//3. 把文档存储到EmbeddingStoreInMemoryEmbeddingStore<TextSegment>embeddingStore=newInMemoryEmbeddingStore<>();EmbeddingStoreIngestor.ingest(documents,embeddingStore);//4. 配置embeddingStore到AiServices里面Assistantassistant=AiServices.builder(Assistant.class).chatModel(BASE_MODEL).chatMemory(MessageWindowChatMemory.withMaxMessages(10)).contentRetriever(EmbeddingStoreContentRetriever.from(embeddingStore)).build();//String chat = assistant.chat("");// 5. 执行问答Stringanswer=assistant.chat("当前活动节点是处理市场与销售的活动，给我推荐几个动词和名词");System.out.println(answer);}

5个步骤;

创建模型
加载文档
把文档存储到EmbeddingStore
配置embeddingStore到AiServices里面
执行问答

publicclassEmbeddingStoreIngestor{privatestaticfinalLoggerlog=LoggerFactory.getLogger(EmbeddingStoreIngestor.class);privatefinalDocumentTransformerdocumentTransformer;privatefinalDocumentSplitterdocumentSplitter;privatefinalTextSegmentTransformertextSegmentTransformer;privatefinalEmbeddingModelembeddingModel;privatefinalEmbeddingStore<TextSegment>embeddingStore;/** * 1. 通过DocumentTransformer转换 documents = documentTransformer.transformAll(documents); * 2. 将documents切割为segments segments = documentSplitter.splitAll(documents); * 3. textSegmentTransformer， segments = textSegmentTransformer.transformAll(segments); * 4. 通过embeddingModel，把segments转换为 List<Embedding> Response<List<Embedding>> embeddingsResponse = embeddingModel.embedAll(segments); * 5. 保存到embeddingStore， embeddingStore.addAll(embeddingsResponse.content(), segments); */publicIngestionResultingest(List<Document>documents){log.debug("Starting to ingest {} documents",documents.size());if(documentTransformer!=null){documents=documentTransformer.transformAll(documents);log.debug("Documents were transformed into {} documents",documents.size());}List<TextSegment>segments;if(documentSplitter!=null){segments=documentSplitter.splitAll(documents);log.debug("Documents were split into {} text segments",segments.size());}else{segments=documents.stream().map(Document::toTextSegment).collect(toList());}if(textSegmentTransformer!=null){segments=textSegmentTransformer.transformAll(segments);log.debug("{} documents were transformed into {} text segments",documents.size(),segments.size());}log.debug("Starting to embed {} text segments",segments.size());Response<List<Embedding>>embeddingsResponse=embeddingModel.embedAll(segments);log.debug("Finished embedding {} text segments",segments.size());log.debug("Starting to store {} text segments into the embedding store",segments.size());embeddingStore.addAll(embeddingsResponse.content(),segments);log.debug("Finished storing {} text segments into the embedding store",segments.size());returnnewIngestionResult(embeddingsResponse.tokenUsage());}}

DocumentSplitter 的本质是把"面向人阅读的连续长文本"转换成"面向向量检索的独立语义单元"，这是连接文档存储和精准检索之间的桥梁。

DocumentByParagraphSplitter DocumentByLineSplitter DocumentBySentenceSplitter DocumentByWordSplitter DocumentByCharacterSplitter DocumentByRegexSplitter Recursive: DocumentSplitters.recursive(...)

Embedding

/** * Embedding.dimension() returns the dimension of the embedding vector (its length) * CosineSimilarity.between(Embedding, Embedding) calculates the cosine similarity between 2 Embeddings * Embedding.normalize() normalizes the embedding vector (in place) * * EmbeddingModel.embed(String) embeds the given text * EmbeddingModel.embed(TextSegment) embeds the given TextSegment * EmbeddingModel.embedAll(List<TextSegment>) embeds all the given TextSegment */@Testvoidtest01(){EmbeddingModelembeddingModel=newBgeSmallEnV15QuantizedEmbeddingModel();Response<Embedding>embeddingResponse=embeddingModel.embed("男人");Embeddingmale=embeddingResponse.content();Response<Embedding>embeddingResponse2=embeddingModel.embed("女人");Embeddingfemale=embeddingResponse2.content();doublebetween1=CosineSimilarity.between(male,female);Response<Embedding>embeddingResponse3=embeddingModel.embed("猿猴");Embeddingmonkey=embeddingResponse3.content();doublebetween2=CosineSimilarity.between(male,monkey);}

Advanced RAG

QueryTransformer: 核心目的是通过改写、扩展或压缩查询，来提高检索质量。 QueryRouter: 决定将处理好的查询（Query）发送给哪一个或哪几个“内容检索器”,其核心目的是确保查询能准确到达相关的数据源，实现精准检索; ContentRetriever: 真正与数据源交互，根据查询去“找出”相关的原始内容。 ContentAggregator： 将来自不同查询、不同检索器的所有“原材料”进行“清洗、融合和精选”。 ContentInjector：将最终聚合好的、高质量的内容，以特定的格式“注入”回用户的原始消息中。

核心的代码逻辑如下： DefaultRetrievalAugmentor

@OverridepublicAugmentationResultaugment(AugmentationRequestaugmentationRequest){ChatMessagechatMessage=augmentationRequest.chatMessage();StringqueryText;if(chatMessageinstanceofUserMessageuserMessage){queryText=userMessage.singleText();}else{thrownewIllegalArgumentException("Unsupported message type: "+chatMessage.type());}QueryoriginalQuery=Query.from(queryText,augmentationRequest.metadata());Collection<Query>queries=queryTransformer.transform(originalQuery);Map<Query,Collection<List<Content>>>queryToContents=process(queries);List<Content>contents=contentAggregator.aggregate(queryToContents);ChatMessageaugmentedChatMessage=contentInjector.inject(contents,chatMessage);returnAugmentationResult.builder().chatMessage(augmentedChatMessage).contents(contents).build();}privateMap<Query,Collection<List<Content>>>process(Collection<Query>queries){if(queries.size()==1){Queryquery=queries.iterator().next();Collection<ContentRetriever>retrievers=queryRouter.route(query);if(retrievers.size()==1){ContentRetrievercontentRetriever=retrievers.iterator().next();List<Content>contents=contentRetriever.retrieve(query);returnsingletonMap(query,singletonList(contents));}elseif(retrievers.size()>1){Collection<List<Content>>contents=retrieveFromAll(retrievers,query).join();returnsingletonMap(query,contents);}else{returnemptyMap();}}elseif(queries.size()>1){Map<Query,CompletableFuture<Collection<List<Content>>>>queryToFutureContents=newConcurrentHashMap<>();queries.forEach(query->{CompletableFuture<Collection<List<Content>>>futureContents=supplyAsync(()->queryRouter.route(query),executor).thenCompose(retrievers->retrieveFromAll(retrievers,query));queryToFutureContents.put(query,futureContents);});returnjoin(queryToFutureContents);}else{returnemptyMap();}}