Loading...
HF多模态

sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco

DistilBert for Dense Passag...

标签:


DistilBert for Dense Passage Retrieval trained with Balanced Topic Aware Sampling (TAS-B)

We provide a retrieval trained DistilBert-based model (we call the dual-encoder then dot-product scoring architecture BERT_Dot) trained with Balanced Topic Aware Sampling on MSMARCO-Passage.

This instance was trained with a batch size of 256 and can be used to re-rank a candidate set or directly for a vector index based dense retrieval. The architecture is a 6-layer DistilBERT, without architecture additions or modifications (we only change the weights during training) – to receive a query/passage representation we pool the CLS vector. We use the same BERT layers for both query and passage encoding (yields better results, and lowers memory requirements).

If you want to know more about our efficient (can be done on a single consumer GPU in 48 hours) batch composition procedure and dual supervision for dense retrieval training, check out our paper: https://arxiv.org/abs/2104.06967 ?

For more information and a minimal usage example please visit: https://github.com/sebastian-hofstaetter/tas-balanced-dense-retrieval


Effectiveness on MSMARCO Passage & TREC-DL’19

We trained our model on the MSMARCO standard (“small”-400K query) training triples re-sampled with our TAS-B method. As teacher models we used the BERT_CAT pairwise scores as well as the ColBERT model for in-batch-negative signals published here: https://github.com/sebastian-hofstaetter/neural-ranking-kd


MSMARCO-DEV (7K)

MRR@10 NDCG@10 Recall@1K
BM25 .194 .241 .857
TAS-B BERT_Dot (Retrieval) .347 .410 .978


TREC-DL’19

For MRR and Recall we use the recommended binarization point of the graded relevance of 2. This might skew the results when compared to other binarization point numbers.

MRR@10 NDCG@10 Recall@1K
BM25 .689 .501 .739
TAS-B BERT_Dot (Retrieval) .883 .717 .843


TREC-DL’20

For MRR and Recall we use the recommended binarization point of the graded relevance of 2. This might skew the results when compared to other binarization point numbers.

MRR@10 NDCG@10 Recall@1K
BM25 .649 .475 .806
TAS-B BERT_Dot (Retrieval) .843 .686 .875

For more baselines, info and analysis, please see the paper: https://arxiv.org/abs/2104.06967


Limitations & Bias

  • The model inherits social biases from both DistilBERT and MSMARCO.

  • The model is only trained on relatively short passages of MSMARCO (avg. 60 words length), so it might struggle with longer text.


Citation

If you use our model checkpoint please cite our work as:

@inproceedings{Hofstaetter2021_tasb_dense_retrieval,
 author = {Sebastian Hofst{\"a}tter and Sheng-Chieh Lin and Jheng-Hong Yang and Jimmy Lin and Allan Hanbury},
 title = {{Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling}},
 booktitle = {Proc. of SIGIR},
 year = {2021},
}

数据统计

数据评估

sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco浏览人数已经达到538,如你需要查询该站的相关权重信息,可以点击"5118数据""爱站数据""Chinaz数据"进入;以目前的网站数据参考,建议大家请以爱站数据为准,更多网站价值评估因素如:sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco的访问速度、搜索引擎收录以及索引量、用户体验等;当然要评估一个站的价值,最主要还是需要根据您自身的需求以及需要,一些确切的数据则需要找sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco的站长进行洽谈提供。如该站的IP、PV、跳出率等!

关于sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco特别声明

本站Ai导航提供的sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco都来源于网络,不保证外部链接的准确性和完整性,同时,对于该外部链接的指向,不由Ai导航实际控制,在2023年5月9日 下午7:16收录时,该网页上的内容,都属于合规合法,后期网页的内容如出现违规,可以直接联系网站管理员进行删除,Ai导航不承担任何责任。

相关导航

暂无评论

暂无评论...