In one of our earlier articles, we shared how PQAI can be used for invalidity searches. A little before that, we had also conducted a benchmarking study to see whether PQAI could actually surface prior art that had already been identified as relevant to an invalidity challenge.

In both cases, we found that PQAI surfaced at least one known prior-art reference. But we did not want to judge PQAI simply based on patents that looked relevant to us. So, at the core of our benchmark, we used patents that had been challenged through Unified Patents’ PATROLL contests, where the winning prior-art submissions were already publicly known.

The idea was simple. If a patent or non-patent-literature reference had independently emerged as a winning submission in a challenge, we could check whether that same reference also surfaced in PQAI. That gave us an external benchmark for assessing how well PQAI was finding known prior art.

During this analysis, we got curious. Were the results from our earlier benchmark and invalidity-search test one-off instances, or would PQAI surface independently identified prior art across other patent challenges too?

So, we decided to take a broader approach to benchmarking PQAI for invalidity searches. In total, we tested it against five public patent challenges, covering 13 known patent references, and checked how many surfaced in PQAI and where they ranked.

Here is what we found.

How We Conducted The Study? 

For this study, we applied the same basic approach across five challenge sets.

We entered the patent claim being investigated into PQAI, applied the relevant date cutoff, which in these tests was the priority date, and examined the initial results. We then checked whether the known winning patent references appeared and recorded their positions.

Where PQAI returned the corresponding publication of the same patent application, we counted it as a surfaced reference. NPL was a different story. In the cases we tested, PQAI was not very effective at surfacing the winning non-patent literature references. Several of those challenge references were standards documents, while the NPL results we saw in PQAI leaned more toward technical papers and research literature. For that reason, the 13-reference comparison below focuses on patent literature.

In some cases, we also used the “More like this” feature to see whether a promising reference could lead us to additional relevant results. However, given the scope of this study, we did not explore those trails in detail. The benchmark below is therefore limited to the initial results PQAI surfaced when we entered the claim under investigation.

Now, let’s look at the five patents and what each result set surfaced.

Benchmark 1: PQAI Surfaced Both Valtrus References

Our first dataset came from the Cloud Native Heroes challenge involving US8379538B2, assigned to Valtrus Innovations.

This was also the case we used for our original PQAI benchmark. We pasted Claim 1 into PQAI and reviewed the first 100 results. Here’s what we found:

Subject Patent	Claim Searched	Known Prior-Art Reference	Surfaced in PQAI?	Position
US8379538B2	Claim 1	US6122664	Yes	#71
US8379538B2	Claim 1	US7454496	Yes	#90

Both of the known patent references appeared in PQAI. However, neither was near the top. Still, this gave us our first useful observation: finding relevant prior art through semantic search may require going well beyond the top results.

 

Benchmark 2: PQAI Surfaced One PulseLink Reference Through Its Published Application

The next challenge involved US7941677B2, owned by PulseLink. Three patent references had been independently identified as prior art as part of the challenge:

• US7565555
• US8046619
• US20070106913

We searched Claim 1 in PQAI and reviewed the first 100 results. Here’s what we found:

Subject Patent	Claim Searched	Known Prior-Art Reference	Surfaced in PQAI?	Position
US7941677B2	Claim 1	US7565555	No	—
US7941677B2	Claim 1	US8046619 via US20090073957A1	Yes	#27
US7941677B2	Claim 1	US20070106913	No	—

PQAI did not surface US8046619 under the exact grant number. However, its corresponding pre-grant publication, US20090073957A1, appeared at rank 27. The other two known references, however, did not appear in the first 100 results.

Benchmark 3: Neither Gaea Patent Reference Surfaced

Our third test involved US12265715B2. The challenge revealed two patent references that we could compare consistently with our PQAI export:

• US9069703
• US10289547

However, neither of the known patent references appeared in the initial results surfaced by PQAI.

Benchmark 4: A Radiant Reference Appeared at Rank 7

The fourth case involved US10404366B2, assigned to Radiant Patents. This one was particularly interesting because we had already used the patent for our practical walkthrough on conducting invalidity searches with PQAI. 

When we ran Claim 1 as input in PQAI, one of the references surfaced at position #7. 

Subject Patent	Claim Searched	Known Prior-Art Reference	Surfaced in PQAI?	Position
US10404366B2	Claim 1	US8805185	Yes	#7
US10404366B2	Claim 1	US6529316	No	—
US10404366B2	Claim 1	JP2000041026	No	—

The other two references, however, were not found.

Benchmark 5: One Aloft Media Reference Appeared at Rank 45

For our fifth test, we used US10372793B2, owned by Aloft Media. This Unified Patents’ PATROLL challenge focused on Claim 23 of the patent. So we gave Claim 23 as input into PQAI and used March 20, 2006, as the relevant search cutoff, and reviewed the first 100 results. Here’s what we found.

Subject Patent	Claim Searched	Known Prior-Art Reference	Surfaced in PQAI?	Position
US10372793B2	Claim 23	US5742768	No	—
US10372793B2	Claim 23	US20040027391A1	Yes	#45
US10372793B2	Claim 23	US20050235203A1	No	—

US20040027391A1 appeared exactly at rank 45. The other two known references did not appear in the first 100 results.

So once again, PQAI surfaced one independently identified reference, but finding it required going beyond the first few results.

Takeaway: PQAI Surfaced Known Prior Art in Four of the Five Challenges

Across the five challenges, PQAI surfaced at least one independently identified patent reference in four cases. At the individual-reference level, it surfaced 5 of the 13 known patent references we checked.

We would not assign an accuracy or success rate based on this, as these 13 references are not an exhaustive set of all potentially relevant prior art. They simply gave us an external benchmark against which we could check PQAI’s results.

However, the bigger takeaway for patent analysts is not to treat the initial ranking as the end of the search. In some tests, the known references appeared much deeper in the results, while others did not surface at all. You can also use features like “More like this” to follow a promising reference into related patents and patent families.

So, if you are using PQAI for conducting an invalidity search, the value is in treating the results as leads to investigate further, not as a complete answer on their own.

How to Use PQAI for Invalidity Searches?

It’s really simple. All you have to do is: 

1. Add the claim being challenged directly into the PQAI search interface.
2. Apply the relevant date cutoff.
3. Review the surfaced results.
4. Use Mapping to understand why a reference surfaced for the claim you shared.
5. Use “More like this” to follow promising references into related patents and patent families.

Want to try it for yourself? Start a PQAI search today by clicking here.