2 years ago

#54955

test-img

sebastienhamel

How to use Lucene ToParentBlockJoinQuery to select a parent only when all conditions of a 'or' child query have been satisfied at least once

I use ToParentBlockJoinQuery in Lucene 7.7.2 which allows to specify conditions on child documents in order to select parent documents. What I am trying to achieve is to use a or condition to select a child, but I want all child query to be satisfied at least once in order to select the parent.

So, if I have:

parent:
    id: "parent-1"
    child:
        id: "child-1"
        number: 20
    child:
        id: "child-2"
        text: "test"
    child:
        id: "child-3"
        text: "some other text with word"
parent:
    id: "parent-2"
    child:
        id: "child-4"
        number: 30
    child:
        id: "child-5"
        text: "test"
parent:
    id: "parent-3"  
    child:
        id: "child-6"
        number: 20
    child:
        id: "child-7"
        text: "test"

Which I could create in code using:


Document parent1 = new Document();
parent1.add(new StringField("id", "parent-1", Field.Store.YES));

Document parent2 = new Document();
parent2.add(new StringField("id", "parent-2", Field.Store.YES));

Document parent3 = new Document();
parent3.add(new StringField("id", "parent-3", Field.Store.YES));

Document child1 = new Document();
child1.add(new StringField("id", "child-1", Field.Store.YES));
child1.add(new IntPoint("number", 20));

Document child2 = new Document();
child2.add(new StringField("id", "child-2", Field.Store.YES));
child2.add(new TextField("text", "test", Field.Store.YES));

Document child3 = new Document();
child3.add(new StringField("id", "child-3", Field.Store.YES));
child3.add(new TextField("text", "some other text with word", Field.Store.YES));

Document child4 = new Document();
child4.add(new StringField("id", "child-4", Field.Store.YES));
child4.add(new IntPoint("number", 30));

Document child5 = new Document();
child5.add(new StringField("id", "child-5", Field.Store.YES));
child5.add(new TextField("text", "test", Field.Store.YES));

Document child6 = new Document();
child6.add(new StringField("id", "child-6", Field.Store.YES));
child6.add(new IntPoint("number", 20));

Document child7 = new Document();
child7.add(new StringField("id", "child-7", Field.Store.YES));
child7.add(new TextField("text", "test", Field.Store.YES));


IndexWriterConfig indexWriterConfig = new IndexWriterConfig(...);
try (IndexWriter writer = new IndexWriter(directory, indexWriterConfig)) {
    // Add the parent-1 block 
    writer.addDocuments(
        List.of(
            child1,
            child2,
            child3,
            parent1
        )
    );

    // Add the parent-2 block 
    writer.addDocuments(
        List.of(
            child4,
            child5,
            parent2
        )
    );

    // Add the parent-3 block 
    writer.addDocuments(
        List.of(
            child6,
            child7,
            parent3
        )
    );
}

With a child query to select a child like this: number <= 20 OR text contains "word"

Which would translate in code to:

// I use a BooleanQuery for each property as there could 
// be more than one clause
BooleanQuery.Builder propertyQuery1 = new BooleanQuery.Builder();
propertyQuery1.add(IntPoint.newRangeQuery("number", 0, 20), BooleanClause.Occur.MUST);

BooleanQuery.Builder propertyQuery2 = new BooleanQuery.Builder();
propertyQuery2.add(new TermQuery(new Term("text", "word")), BooleanClause.Occur.MUST);

// This is the 'or' query mentioned in the question
BooleanQuery.Builder childQuery = new BooleanQuery.Builder();
childQuery.setMinimumNumberShouldMatch(1);
childQuery.add(propertyQuery1.build(),  BooleanClause.Occur.SHOULD);
childQuery.add(propertyQuery2.build(),  BooleanClause.Occur.SHOULD);

It would select parent-1 and parent-3, since both of them contains a child that satisfies the childQuery. (This is what is implemented in the code below)

Now, the condition I want to add should specify that every child query should be satisfied at least once. Meaning, in order to return a parent, I should have at least one child satisfying the first condition (number <= 20) AND at least one child satisfying the second condition (text contains "word").

In this case, only parent-1 would be selected as every conditions are satisfied by at least one child, child-1 satisfies number <= 20 and child-3 satisfies text contains "word". parent-2 would not be returned as it does not contain a child for which text contains "word" condition is true.

So, using the child query already defined, this is the code now:

// first create the query that selects the parent based on the childQuery already defined...
ToParentBlockJoinQuery childJoinQuery =
                    new ToParentBlockJoinQuery(childQuery.build(), parentsFilter, ScoreMode.Avg);

BooleanQuery.Builder fullQuery = new BooleanQuery.Builder();

fullQuery.add(new BooleanClause(childJoinQuery, BooleanClause.Occur.MUST));
fullQuery.add(new BooleanClause(new MatchAllDocsQuery(), BooleanClause.Occur.MUST));

TopDocs topDocs = searcher.search(fullQuery.build(), 10);

// I need to return the children that satistifed the child query
// along with the parent document 
List<Pair<Document, List<Document>>> documents = new ArrayList<>();
for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
    val doc = searcher.doc(scoreDoc.doc);

    List<Document> childrenDocuments = new ArrayList<>();

    // find matching children
    ParentChildrenBlockJoinQuery childrenQuery =
            new ParentChildrenBlockJoinQuery(parentsFilter, childQuery.build(), scoreDoc.doc);
    TopDocs matchingChildren = searcher.search(childrenQuery, topChildrenHits);

    for (ScoreDoc childScoreDoc : matchingChildren.scoreDocs) {
        val child = searcher.doc(childScoreDoc.doc);
        childrenDocuments.add(child);
    }

    documents.add(Pair.of(doc, childrenDocuments));
}
return documents;

When iterating over the children, I could test for each property query and make sure all property queries have been satisfied at least once, but it screws up the top n hits for query, since I will have to discard results from the 10 received. To fix the last problem, I could send as much requests as needed in order to fulfill the required top n hits. The last option could work but, I worry about performance: a full query handled once by Lucene surely would be more performant.

Any idea?

java

search

lucene

0 Answers

Your Answer

Accepted video resources