DGrid AI has introduced a new Proof of Quality (PoQ) framework designed to evaluate AI outputs and improve reward distribution across decentralized networks. According to the company's latest research paper, the framework addresses a core flaw in existing quality scoring systems that have relied on having correct answers to compare against. The new system uses small evaluator models to score each output's quality, with these scores driving the rewards, creating a cost-effective and scalable solution for decentralized AI networks. As reported by DGrid AI, the framework represents the fourth paper in their ongoing research series on Proof of Quality, building upon incremental improvements that include latency-adjusted payouts, defenses against manipulative scorers, and more granular breakdown of quality metrics.
The research introduces three specialized AI judges trained specifically for reference-free quality scoring, each taking a question and response as input and outputting scores from 0 to 10 without access to correct answers. As reported by DGrid AI, the judges differ primarily in size and speed: TextCNN (~10M parameters) runs in approximately 1 millisecond per call, MiniLM (22M parameters) takes around 13 milliseconds, and DeBERTa (184M parameters) requires roughly 15 milliseconds. The judges were trained using a two-stage process, first pre-trained on UltraFeedback dataset before fine-tuning on the network's own task distribution. The training approach was designed to give judges a broad baseline understanding of quality before narrowing their focus to the specific scoring context.
On a held-out test set of 300 examples, the DeBERTa judge achieved a Pearson correlation of 0.747 against ground-truth proxy without access to any reference answer, significantly outperforming the reference-based evaluators from the prior framework that reached a maximum of 0.647. According to the research, the performance gap reflects that the new judges were optimized end-to-end for the scoring task itself, compared to the older similarity metrics that measured cosine distance to reference embeddings. The system includes deployment-oriented features such as a cascading pipeline that reduces evaluation costs by up to 72.7% at aggressive threshold settings, though correlation drops to around 0.51 in that configuration. An online calibration mechanism runs without manual tuning, consistently identifying semantic quality as the dominant signal and adjusting weights accordingly, assigning it 4.7 times its starting weight over time.
The judges perform unevenly across different task types, achieving 0.830 correlation on question answering but falling to 0.199 on summarization. As reported by DGrid AI, the authors attribute this performance gap to the evaluation metric used during training, noting that raw word overlap is a poor measure of summarization quality. The paper describes this as the primary open problem rather than a known limitation being managed quietly, with the authors presenting methodical results that include both improvements and failure cases. The authors emphasize that the judges correlate well with token-level word overlap as a ground truth proxy, but whether word overlap reliably reflects what a human would consider a quality response remains an unresolved question.
