ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Notes:

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a software package for automatically evaluating the quality of summaries. It is commonly used in the field of natural language processing (NLP) to evaluate the performance of automatic summarization algorithms.

ROUGE works by comparing an automatically generated summary with a reference summary (i.e. a summary that has been written by a human) and computing a score that reflects the degree of overlap between the two summaries. ROUGE uses a variety of metrics to measure this overlap, including:

ROUGE-N: This metric measures the overlap between the automatically generated summary and the reference summary at the word level, using n-grams (i.e. sequences of n consecutive words). For example, ROUGE-1 measures the overlap between the summaries at the unigram (i.e. single word) level, while ROUGE-2 measures the overlap at the bigram (i.e. two-word) level.
ROUGE-L: This metric measures the longest common subsequence (LCS) between the automatically generated summary and the reference summary. The LCS is the longest sequence of words that appears in both summaries, in the same order. ROUGE-L is considered to be a more sophisticated metric than ROUGE-N, because it takes the order of words into account, whereas ROUGE-N does not.
ROUGE-W: This metric is similar to ROUGE-N, but it weights each word in the summaries according to its frequency in the reference summary. Words that appear more frequently in the reference summary are given more weight, whereas words that appear less frequently are given less weight. This helps to account for the fact that some words may be more important than others in the context of a summary.

In summary, ROUGE is a software package for automatically evaluating the quality of summaries. It compares an automatically generated summary with a reference summary and computes a score that reflects the degree of overlap between the two summaries. ROUGE uses a variety of metrics, including ROUGE-N, ROUGE-L, and ROUGE-W, to measure this overlap. These metrics are commonly used in the field of NLP to evaluate the performance of automatic summarization algorithms.

Resources:

github.com/ng-j-p/rouge-we .. software package for automated evaluation of summaries

Wikipedia:

References:

Automatic Summarization (2011)

Assessing the factual accuracy of generated text
B Goodrich, V Rao, PJ Liu, M Saleh – Proceedings of the 25th ACM …, 2019 – dl.acm.org
… ABSTRACT We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy) …

Small-world networks for summarization of biomedical articles
M Moradi – arXiv preprint arXiv:1903.02861, 2019 – arxiv.org
… Recall-Oriented Understudy for Gisting Evaluation (ROUGE) toolkit. The results show that the degree can … 11 S25 16 24 S75 15 12 S48 16 25 S14 14 13 S63 16 26 S80 13 We use the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) toolkit [14] in our experiments …

Assessing The Factual Accuracy of Text Generation
B Goodrich, MA Saleh, P Liu, V Rao – 2019 – research.google
… Abstract. We propose an automatic metric to reflect the factual accuracy of generated text as an alternative to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy) …

VIKOR Algorithm Based on Cuckoo Search for Multi-document Text Summarization
ZH Ali, AA Noor, MA Jassim – … on Applied Computing to Support Industry …, 2019 – Springer
… documents. The performance of the proposed model is evaluated using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric. The obtained results support the effectiveness of the proposed model. Keywords …

Fuzzy AHP approach for legal judgement summarization
N Bansal, A Sharma, RK Singh – Journal of Management …, 2019 – orsociety.tandfonline.com
… CaseSummarizer used dates, named entities, and proximity to section headings for computing sentence scores. Authors evaluated their approach using Recall-Oriented Understudy for Gisting Evaluation (ROUGE) tool and human scoring system …

Comparison of automatic methods for reducing the Pareto front to a single solution applied to multi-document text summarization
JM Sanchez-Gomez, MA Vega-Rodríguez… – Knowledge-Based …, 2019 – Elsevier
… The methods have been compared with Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics … All methods have been evaluated and compared by using Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics [18] …

Automatic Text Generation in Macedonian Using Recurrent Neural Networks
I Milanova, K Sarvanoska, V Srbinoski… – … Conference on ICT …, 2019 – Springer
… In order to measure how closely the generated text resembles a human written text, we used a metric called ROUGE-N (Recall-Oriented Understudy for Gisting Evaluation), which is a set of metrics for evaluating automatic generation of texts as well as machine translation …

Extending The Performance of Extractive Text Summarization By Ensemble Techniques
A Bharadwaj, A Srinivasan, A Kasi… – 2019 11th International …, 2019 – ieeexplore.ieee.org
… The quality of the summary generated is evaluated based on the Recall Oriented Understudy for Gisting Evaluation (ROUGE) metric … In this work, the Recall- Oriented Understudy for Gisting Evaluation (ROUGE) [10] package is used for performing quantitative assessment …

Wajeez: An Extractive Automatic Arabic Text Summarisation System
A Al Oudah, K Al Bassam, H Kurdi… – … Conference on Human …, 2019 – Springer
… Wajeez was applied on two different datasets: the Essex Arabic Summaries Corpus (EASC) and a manual summary to assess its performance using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) set of metrics …

Enhanced continuous and discrete multi objective particle swarm optimization for text summarization
V Priya, K Umamaheswari – Cluster Computing, 2019 – Springer
… system. The performance of the system is calculated using the ROUGE-N metric. ROUGE is Recall Oriented Understudy for Gisting Evaluation [28] used as performance metric to evaluate the quality of the summary generation. It …

Albayzin Evaluation: IberSPEECH-RTVE 2018 Speech to Text Transcription Challenge
E Lleida, A Ortega, A Miguel, V Bazán, C Pérez… – catedrartve.unizar.es
… measures as Word Information Loss [2], which is more suitable than WER for the evaluation of any application in which the proportion of word information communicated is more meaningful than edit cost or Recall-Oriented Understudy for Gisting Evaluation (ROUGE)[3] widely …

Multilingual Text Summarization based on LDA and Modified PageRank
S Malallah, ZH Ali – Iraqi Journal of Information Technology, 2019 – iasj.net
… Page 2. 140 Iraqi Journal of Information Technology. V.9 N.3. 2019 2018 Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric. The obtained results support the effectiveness of the proposed model. Keywords: LDA, PageRank, ROUGE, TAC-2011 ???????? … ?

Towards Neural Similarity Evaluator
H Kané, Y Kocyigit, P Ajanoh, A Abdalla, M Coulibali – 2019 – openreview.net
… 2.1 BLEU, ROUGE and n-gram matching approaches BLEU (Bilingual Evaluation Understudy) [Papineni et al., 2002] and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [Lin, 2004] have been used to evaluate many NLP tasks for almost two decades …

Extractive Text Summarization Using Ontology and Graph-Based Method
C Yongkiatpanich, D Wichadakul – 2019 IEEE 4th International …, 2019 – ieeexplore.ieee.org
… using 400 biological review papers as a corpus randomly sampled from PubMed Central. Our approach outperformed the baseline comparators in terms of Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scores …

Extractive document summarization using an adaptive, knowledge based cognitive model
M Rajangam, C Annamalai – Cognitive Systems Research, 2019 – Elsevier
… The performance of the proposed cognitive model was compared with the existing state-of-art methods using the DUC (Document Understanding Conference) 2001 dataset and the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) summary evaluation tool …

Multi-document extractive text summarization: A comparative assessment on features
B Mutlu, EA Sezer, MA Akcayol – Knowledge-Based Systems, 2019 – Elsevier
… The summarization performances of these models were evaluated using original classification performance metrics, and recall-oriented understudy for gisting evaluation (ROUGE)-n. This study recommended the use of fuzzy systems based on a feature vector and a fuzzy rule …

Recurrent Convolution Attention Model (RCAM) for Text Generation based on Title
Y Jianglin, G Zhigang, C Gang – Journal of Physics: Conference …, 2019 – iopscience.iop.org
… ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [16] is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The …

Red-faced ROUGE: Examining the Suitability of ROUGE for Opinion Summary Evaluation
W Tay, A Joshi, XJ Zhang, S Karimi, S Wan – Proceedings of the The …, 2019 – aclweb.org
… 2009). We focus on the most frequently reported metric for opinion summari- sation, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004) and leave the analysis of other evaluation metrics to future stud- ies …

A Novel Approach of Text Summarization using Assamese WordNet
N Baruah, SK Sarma… – 2019 4th International …, 2019 – ieeexplore.ieee.org
… The proposed approach of this paper has been evaluated using ROUGE (Recall-Oriented Understudy for Gisting Evaluation) as a standard metric for evaluation in most of the summarization conferences. With respect to content …

Modeling Health Coaching Dialogues for Behavioral Goal Extraction
I Gupta, B Di Eugenio, B Ziebart, B Liu… – 2019 IEEE …, 2019 – ieeexplore.ieee.org
… We also evaluated our results using Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [16], a well known metric for summarization and machine translation and achieved an F- score of 0.57 using ROUGE over unigrams. Though it’s not the …

Deep Learning Architecture for Multi-Document Summarization as a cascade of Abstractive and Extractive Summarization approaches
AK Singh, M Shashi – International Journal of Computer Sciences …, 2019 – researchgate.net
… Recall-Oriented Understudy for Gisting Evaluation [10] or ROUGE is a Recall based metric which is used to evaluate fixed length summaries making use of n-gram co- occurrence of words/phrases in summaries generated with respect to Reference summaries generated by …

Graph-based text summarization using modified TextRank
C Mallick, AK Das, M Dutta, AK Das… – Soft Computing in Data …, 2019 – Springer
… 1.00000. 0.67647. 0.55232. 1.00000. 0.70561. 0.62140. 1.00000. 0.66236. 0.40092. 1.00000. 0.60510. 3 Experimental Result. Since early 2000s, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [9] is widely used for performance evaluation of summarization techniques …

A Better Gauging Model for the Evaluation of Automatic Machine Translation of English–Hindi Language
P Malik, Y Mrudula, AS Baghel – pdfs.semanticscholar.org
… GTM outperforms BLEU and NIST in correlation with human scores [23], no matter what the available number of reference translations is.Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [24] is quite similar to the BLEU metric, but unlike BLEU which is precision …

Automatic Meta-evaluation of Low-Resource Machine Translation Evaluation Metrics
J Yu, W Liu, H He, L Wang – 2019 International Conference on …, 2019 – ieeexplore.ieee.org
… relationship between long-distance discontinuous words. In this respect, in 2004, Lin proposed a method named Recall-Oriented Understudy for Gisting Evaluation (ROUGE) based on discontinuous n-grams to describe the …

Development of a Konkani Language Dataset for Automatic Text Summarization and its Challenges
J D’Silva, U Sharma – ripublication.com
… An effective tool that can aid in achieving this purpose is the ROUGE package which is a short form of “Recall-Oriented Understudy for Gisting Evaluation”. ROUGE is one of the ways to compute the effectiveness of auto generated summaries …

A Survey on Evaluation Methods for Chatbots
W Maroengsit, T Piyakulpinyo, K Phonyiam… – Proceedings of the …, 2019 – dl.acm.org
… In a way, precision and recall are the basic fundamentals of automatic evaluation to the like of BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall- Oriented Understudy for Gisting Evaluation). Table 2. Chatbot Evaluation Methods …

An analysis of templates for generating text for use in comparing with data-driven models
M Poulter – projects.cs.uct.ac.za
… 5.1 Metric evaluation The E2E NLG Challenge currently uses five metrics to evaluate the submitted NLG systems with [20]. This paper looks at two of these, namely Bilingual Evaluation Understudy (BLEU) and Recall Oriented Understudy for Gisting Evaluation (ROUGE) …

Corpora and Evaluation for Text Summarisation
P Mehta, P Majumder – From Extractive to Abstractive Summarization: A …, 2019 – Springer
… BLEU uses n-gram overlap to compute similarity between expected and actual summaries. 3.3.3 ROUGE Measure. ROUGE [11] or Recall-Oriented Understudy for Gisting Evaluation is another extremely popular automatic evaluation measure that has been derived from BLEU …

Evaluating Machine Translation Systems using Weighted Vote of Different Scoring Metrics
A Sharma, A Sharifian, S Kumar – trane293.github.io
… 2.2 ROGUE-L Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004) is a set of metric used for evaluating automatic summarization and machine translation software in natural language processing. ROGUE …

A Survey on Evaluation Methods for Chatbots [Draft Version]
W Maroengsit, S Pongnumkul, T Piyakulpinyo… – researchgate.net
… percentage. In a way, precision and recall are the basic fundamentals of automatic evaluation to the like of BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation). Precision …

Summarizing social media content for multimedia stories creation
F Amato, F Moscato, V Moscato, A Picariello – 2019 – ceur-ws.org
… Dataset Vertices Edges Users Topic labels Images YFCC100M 1K 40 1.3K 3.8K Table 1. Topic-based view characterization We performed a human-based evaluation for generated summaries using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE 5) package …

Highlighted Word Encoding for Abstractive Text Summarization
DM Lal, KP Singh, US Tiwary – International Conference on Intelligent …, 2019 – Springer
… 5.2 Experimental Setup. Open image in new window. 5.3 Evaluation Metric. ROUGE [14], an acronym for Recall-Oriented Understudy for Gisting Evaluation, is a package used to evaluate the quality of the machine-generated summaries against a human-generated summary …

EASY-M: Evaluation System for Multilingual Summarizers
M Litvak, N Vanetik, Y Veksler – … ACROSS LANGUAGES, GENRES AND …, 2019 – aclweb.org
… Metrics in the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) family, proposed in (Lin, 2004), count the number of overlapping units such as n-grams, word sequences, and word pairs between the system and the reference sum- maries …

Learning to Generate Questions with Adaptive Copying Neural Networks
X Lu – Proceedings of the 2019 International Conference on …, 2019 – dl.acm.org
… ROUGE (Recall- Oriented Understudy for Gisting Evaluation) measures the co-occurrences between the system-generated summary and the content in a human-generated summary. ROUGE-L mea- sures the co-occurrences of the longest common subsequence …

A Multi-document Summarization System for News Articles in Portuguese using Integer Linear Programming
LB de Brito Gomes, HTA de Oliveira… – bracis2019.ufba.br
… al. 2014]. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [Lin 2004] measures were adopted to evaluate the generated summaries in comparison with human-made references summaries. The experimental …

Central embeddings for extractive summarization based on similarity
SJ Gutiérrez Hinojosa, H Calvo… – Computación y …, 2019 – cys.cic.ipn.mx
… 3]. 2.3.1 Recall-Oriented Understudy for Gisting Evaluation (ROUGE) The University of Southern California’s Information Sciences Institute (ISI) developed the recall-based metric called ROUGE-N defined by Equation 1 [15] …

Extractive Text Summarization Using Graph Based Ranking Algorithm And Mean Shift Clustering
R Ramesh, B Rajan – Available at SSRN 3439357, 2019 – papers.ssrn.com
… ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It works by comparing an automatically produced extractive summary known as system generated summary or translation against a reference summary known as gold standard (typically human-produced) …

Implementation of Chinese Reader Aid for Visually-Impaired by Using Neural Network and Text Summarization Technologies
LJ Chen, CY Chen, HY Chen… – … on Machine Learning …, 2019 – ieeexplore.ieee.org
… summary. 2) Performance Evaluation: Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is used for evaluating automatic summarization and machine translation software in natural language processing. ROUGE compares …

Extractive Summarization of a Document Using Lexical Chains
C Mallick, M Dutta, AK Das, A Sarkar… – Soft Computing in Data …, 2019 – Springer
… 0.47660. 0.45560. 0.40736. 0.41649. Evaluation using Rouge: Since the early 2000s Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [30] is widely used for automatic summaries and also for the performance evaluation. For the first time, it is used in DUC 2004 …

Overview and results: Cl-scisumm shared task 2019
MK Chandrasekaran, M Yasunaga, D Radev… – arXiv preprint arXiv …, 2019 – arxiv.org
… We have been reporting ROUGE scoring since CL-SciSumm 17, for Tasks 1a and Task 2. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics used to automatically evaluate summarization systems [11] by measuring the overlap between computer …

Derive Summary Using Latent Factors
S Umamaheswaran – 2019 10th International Conference on …, 2019 – ieeexplore.ieee.org
… In [10] a study on Recall-Oriented Understudy for Gisting Evaluation (ROUGE) shows that it can be widely used for automatic summaries and also for the performance evaluation since the early 2000s. ROUGE_N N-gram provides cooccurrence statistics …

Automatic Slide Generation for Scientific Papers
A Sefid, J Wu, P Mitra, CL Giles – 2019 – sciknow.github.io
… 4.3 Evaluation Metric The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [10] is a standard metric for automatic evaluation of machine- generated summaries. ROUGE-N is an n-gram overlap between a candidate summary and a reference summary …

Question answering as an automatic evaluation metric for news article summarization
M Eyal, T Baumel, M Elhadad – arXiv preprint arXiv:1906.00318, 2019 – arxiv.org
… to a query (Hobson et al., 2007). ROUGE, or “Recall-Oriented Understudy for Gisting Evaluation” (Lin, 2004), refers to a set of automatic intrinsic metrics for evaluating au- tomatic summaries. ROUGE-N scores a candi- date summary …

Intelligent multi-document summarization for biomedical literature by word embeddings and graph-based ranking
C Shen, H Lin, H Hao, Z Yang… – Journal of Intelligent …, 2019 – content.iospress.com
… For evaluation, we use Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [15], which is widely used evaluation method in text summarization. It is proposed by Lin [16] and adopted by the automatic summary evaluation task in the TREC (Text Retrieval Conference) …

Automatic text summarization of Swedish news articles
N Lehto, M Sjödin – 2019 – diva-portal.org
… [23] ROUGE Recall-Oriented Understudy for Gisting Evaluation, or ROUGE for short, is a package created with the purpose to automatically determine the quality of an automatically generated text summary in comparison to a reference gold standard summary …

Concisenet: An End To End Abstractive Model For Topic Generation
SRM IST-KTR, SRM IST–KTR – pdfs.semanticscholar.org
… Recall Oriented Understudy for Gisting Evaluation or ROUGE is an evaluation metric for specifically evaluating Text Summarization and Machine Translation application on the basis of metrics such as Precision and Recall. Recall …

SummCoder: An unsupervised framework for extractive text summarization based on deep auto-encoders
A Joshi, E Fidalgo, E Alegre… – Expert Systems with …, 2019 – Elsevier
… Empirical results show that, on DUC 2002, on Blog Summarization, and on TIDSumm datasets, our text summarization approach obtains comparable or better performance than the state-of-the-art methods for different ROUGE metrics …

Video description: A survey of methods, datasets, and evaluation metrics
N Aafaq, A Mian, W Liu, SZ Gilani, M Shah – ACM Computing Surveys …, 2019 – dl.acm.org
… translation and image captioning metrics, including Bilingual Evaluation Understudy (BLEU) [102], Recall Oriented Understudy for Gisting Evaluation (ROUGE) [82], Metric for Evaluation of Translation with Explicit Ordering (METEOR) [12], Consensus-based Image Description …

A Rewriter Model for Urdu Document Concision with Neural Word Embeddings
M Shahid, S Saleem, A Dilawari… – Urdu News Headline …, 2019 – researchgate.net
… Fig. 2. Loss Graph showing the progress of training VII. RESULTS Recall Oriented Understudy for Gisting Evaluation (ROUGE) is one of a metric which is used for the evaluation of the computer-generated summaries with the reference summaries generated by the humans …

From Web Crawled Text to Project Descriptions: Automatic Summarizing of Social Innovation Projects
G Nenadic – … : 24th International Conference on Applications of …, 2019 – books.google.com
… ROUGE or Pyramid. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a commonly used metric in summarization literature [5] that is based on overlapping n- grams in summary and original text. There are several …

A Supervised Approach For Extractive Text Summarization Using Minimal Robust Features
D Krishnan, P Bharathy… – … on Intelligent Computing …, 2019 – ieeexplore.ieee.org
… V. EVALUATION METRIC ROUGE can be elaborated as Recall-Oriented Understudy for Gisting Evaluation. It’s a set of metrics mainly used for evaluating summarization of texts. The evaluation is done by comparing the quality …

A Supervised Method for Extractive Single Document Summarization based on Sentence Embeddings and Neural Networks
S Lamsiyah, A El Mahdaouy… – AI2SD’2019 …, 2019 – hal.archives-ouvertes.fr
… To train our method we have used the 200-word extractive gold standard summaries. For evaluation, we adopted the Recall-oriented Understudy for Gisting Evaluation (ROUGE) [17], which is a fully automated and the state-of-the-art method for text summa- rization evaluation …

Two-Level Text Summarization with Natural Language Processing
R Hande, A Sidhwani, D Sidhwani, M Shiv… – … on Computer Networks …, 2019 – Springer
… 4 Evaluation of Summaries. The summaries are evaluated using ROUGE Matrix, which stands for Recall-Oriented Understudy for Gisting Evaluation [4]. ROUGE is a human-based evaluation of summaries. In this the summary …

Translate and Summarize Complaints of Patient to Electronic Health Record by BiLSTM-CNN Attention model
H Song, G Li, S Hou, YY Qu… – 2019 12th International …, 2019 – ieeexplore.ieee.org
… There are some exiting criterions have been formulated for evaluate text summarization like SUMMAC, DUC, ROUGE and so on. In this project, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is chosen as the quantity’s evaluation method …

A Graph Based Approach on Extractive Summarization
M Dutta, AK Das, C Mallick, A Sarkar… – Emerging Technologies in …, 2019 – Springer
… Lin introduced a set of metrics called Recall-Oriented Understudy for Gisting Evaluation (ROUGE) to automatically determine the quality of a summary by comparing it to reference summaries developed by humans which is generally considered as ground truth …

Predicting commentaries on a financial report with recurrent neural networks
K El Mokhtari, J Maidens, A Bener – Canadian Conference on Artificial …, 2019 – Springer
… In both experiments, we use the following evaluation metrics: BLEU (BiLingual Evaluation Understudy) [17], ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [18] and METEOR (Metric for Evaluation of Translation with Explicit ORdering) [19] …

Abstractive Text Summarization Using Artificial Intelligence
C Parmar, R Chaubey, K Bhatt… – … Conference on Advances …, 2019 – papers.ssrn.com
… 3. ROUGE ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is essentially a set of metrics for evaluating machine generated summaries of texts. It works by comparing automatically produced summary against a set of reference summaries [15] …

From web crawled text to project descriptions: automatic summarizing of social innovation projects
N Miloševi?, D Marinov, A Gök, G Nenadi? – International Conference on …, 2019 – Springer
… Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a commonly used metric in summarization literature [5] that is based on overlapping n-grams in summary and original text. There are several variants of ROUGE …

Utilize Discourse Relations to Segment Document for Effective Summarization
L Jiazheng, M Rafi – … on Semantics, Knowledge and Grids (SKG …, 2019 – ieeexplore.ieee.org
… D. Evaluate the performance of summaries ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [19] is a common metric to evaluate automatically the performance of a generated summary. ROUGE-N ( ? {1,2}) represents the n-gram units used to evaluate how well …

Keyword Extraction Performance Analysis
A Kumbhar, M Savargaonkar… – … IEEE Conference on …, 2019 – ieeexplore.ieee.org
… C. Evaluation Metrics Three evaluation metrics viz. precision, recall and F- measure (F1) are employed for evaluating the performance of methods [17]. We have used Recall-Oriented Understudy for Gisting Evaluation (ROUGE 2.0) method [18]. IV …

Thai Scene Graph Generation from Images and Applications
P Khuphiran, S Kajkamhaeng… – 2019 23rd International … – ieeexplore.ieee.org
… stemming and synonym matching. ROUGE, (Recall-Oriented Understudy for Gisting Evaluation), remod- els from BLEU adding more attention to recall than pre- cision by paying attention to N-gram. CIDEr (Consensus- based Image …

Abstractive text summarization using LSTM-CNN based deep learning
S Song, H Huang, T Ruan – Multimedia Tools and Applications, 2019 – Springer
… 3.3 Evaluation method. In our work, we use the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) toolkit [13], which has been widely adopted by DUC for automatic summarization evaluation. ROUGE measures summary …

A new automatic multi-document text summarization using topic modeling
RK Roul, S Mehrotra, Y Pungaliya… – International conference on …, 2019 – Springer
… summaries. DUC-2005 has 50 and DUC-2007 has 45 document sets. ROUGE or Recall-Oriented Understudy for Gisting Evaluation score [11] is used by the approach to measure the performance of text summarization. ROUGE …

Automatic Text Summarization of News Articles in Serbian Language
D Kosmajac, V Kešelj – 2019 18th International Symposium …, 2019 – ieeexplore.ieee.org
… V. RESULTS A. Evaluation Metrics Recall-oriented Understudy for Gisting Evaluation (ROUGE) is a set of evaluation metrics introduced by Lin [21] to automat- ically evaluate the quality of a summary. ROUGE is adopted evaluation …

Automating News Summarization with Sentence Vectors Offset
M Steinert, R Granada, JP Aires… – 2019 8th Brazilian …, 2019 – ieeexplore.ieee.org
… C. ROUGE Scores Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [13] is a metric used for evaluating automatic summarization, ie, for automatically determining the quality of a summary. It compares a given summary …

A Hierarchical Neural Summarization Framework for Spoken Documents
TE Liu, SH Liu, B Chen – ICASSP 2019-2019 IEEE International …, 2019 – ieeexplore.ieee.org
… 8, pp. 2551–2590, 2007. [22] CY Lin, “ROUGE: Recall-oriented Understudy for Gisting Evaluation,” 2003 [Online]. Available: http://haydn.isi.edu/ROUGE/. [23] Y. Gong and X. Liu, “Generic Text Summarization Using Relevance Measure and Latent Semantic Analysis,” in Proc …

Deep Learning Based Extractive Text Summarization: Approaches, Datasets and Evaluation Measures
D Suleiman, AA Awajan – 2019 Sixth International Conference …, 2019 – ieeexplore.ieee.org
… TABLE II DATASETS OF EXTRACTIVE SUMMARIZATION IV. EVALUATION MEASURES Extractive text summarization is evaluated using a package called ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [23]. ROUGE …

SRL-ESA-TextSum: A text summarization approach based on semantic role labeling and explicit semantic analysis
M Mohamed, M Oussalah – Information Processing & Management, 2019 – Elsevier
… Experimental results indicate that the proposed summarizer outperforms all state-of-the-art related comparators in the single document summarization based on the ROUGE-1 and ROUGE-2 measures, while also ranking second in the ROUGE-1 and ROUGE-SU4 scores for the …

Towards Guided Summarization of Scientific Articles: Selection of Important Update Sentences
GH Rachman, ML Khodra… – … Conference on Electrical …, 2019 – ieeexplore.ieee.org
… The last that automated update summary is evaluated by utilizing ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [16] with update summary that has been created manually. We use ROUGE 2.0 library2 from [17] …

A Comparative Study of Deep Learning Approaches for Query-Focused Extractive Multi-Document Summarization
T Sakai – 2019 IEEE 2nd International Conference on …, 2019 – ieeexplore.ieee.org
… Specifically, given a sequence of sentences S = [s1, s2, s3, …, sn] and a sequence of human-generated summaries M = [m1, m2, …, mn], we give a score to each sentence based on some variants of ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [8]: ROUGE …

Semantic Variance Measurements For Evaluating Single-Document Summarization
ET AL-SHAMMARI – Journal of Theoretical and Applied Information …, 2019 – jatit.org
… the summary. Another approach used for text summarization is the use of Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004) is a software package for automatic analysis of summaries. The evaluation approach …

Attention-Based Deep Learning Model for Image Captioning: A Comparative Study
PP Khaing – International Journal of Image, Graphics and Signal …, 2019 – mecs-press.net
… The commonly used evaluation metrics are Bilingual Evaluation Understudy (BLEU) [25], Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [26], Metric for Evaluation based Image Description Evaluation (METEOR) [27], and Consensus- based Image Description …

Regularizing output distribution of abstractive chinese social media text summarization for improved semantic consistency
B Wei, X Ren, Y Zhang, X Cai, Q Su, X Sun – ACM Transactions on …, 2019 – dl.acm.org
… Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [16] is widely used for summarization evaluation. However, as ROUGE is designed for extractive text summarization, it cannot deal with summary paraphrasing in abstractive text summarization …

Context-Based Multi-document Summarization
S Sonawane, A Ghotkar, S Hinge – Contemporary Advances in Innovative …, 2019 – Springer
… DUC 2002 dataset contains total 60 topics and each topic has 10 documents. The results have been evaluated using ROUGE evaluation toolkit [21], which is a n-gram based method. ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation …

Predicting Commentaries On A Financial Report With Recurrent Neural Networks
A Bener – johnmaidens.com
… In both ex- periments, we use the following evaluation metrics: BLEU (BiLingual Evaluation Understudy) [17], ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [18] and METEOR (Metric for Evaluation of Translation with Explicit ORder- ing) [19] …

Deep Reinforced Self-Attention Masks for Abstractive Summarization (DR. SAS)
A Chadha, M Masoud – arXiv preprint arXiv:2001.00009, 2019 – arxiv.org
… Quantitatively, we evaluate our method in terms of ROUGE. ROUGE stands for Recall- Oriented Understudy for Gisting Evaluation. It is a set of metrics for evaluating automatic summarization of texts as well as machine translation …

Assessing the semantic space bias caused by ASR error propagation and its effect on spoken document summarization
MA Tündik, V Kaszás, G Szaszák – Proc. Interspeech 2019, 2019 – isca-speech.org
… We hence prepare summaries for the MT-MP, AT-MP, MT- AP and AT-AP scenarios and compare them based on the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric family [28]. ROUGE is one of the most commonly used benchmark in text summarization …

Extractive summarization of clinical trial descriptions
C Gulden, M Kirchner, C Schüttler, M Hinderer… – International journal of …, 2019 – Elsevier
… 2.4. Calculating ROUGE-Scores for the summaries. We adopt the widely used Recall-Oriented Understudy for Gisting Evaluation or ROUGE [26] metrics for evaluating the automatically generated summaries. ROUGE measures …

Oboyob: A sequential-semantic Bengali image captioning engine
T Deb, MZA Ali, S Bhowmik, A Firoze… – Journal of Intelligent …, 2019 – content.iospress.com
… underestimated in some cases. 6.2.3Recall-Oriented Understudy for Gisting Evaluation (ROUGE). We additionally involve state-of-the-art ROUGE L [33], a measure based on the Longest Common Subsequence (LCS). The score is …

Automatic Summarization of Technical Documents in the Oil and Gas Industry
JMC Marques, FG Cozman… – 2019 8th Brazilian …, 2019 – ieeexplore.ieee.org
… While there exist many human-based strategies for assessing the quality of an auto- matically generated abstract, such as the ones used by Parveen and Stube [5], the most used metric in the field is the Recall- Oriented Understudy for Gisting Evaluation (ROUGE) [3], [6] …

PSO-Based Text Summarization Approach Using Sentiment Analysis
S Mandal, GK Singh, A Pal – Computing, Communication and Signal …, 2019 – Springer
… To evaluate the performance of the model with user-generated summary, F-measure is used. F-measure uses the generic metric and ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation use to compare the quality of summary …

Ranking sentences from product description & bullets for better search
P Verma, A Kutiyanawala, K Shen – arXiv preprint arXiv:1907.06330, 2019 – arxiv.org
… reinforcement learning (described in 3.2). ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is a metric to compare automatically generated sum- mary with the reference summary. ROUGE makes use of …

Convolutional Neural Network based for Automatic Text Summarization
WH Alquliti, NBA Ghani – researchgate.net
… The ROUGE (Recall Oriented Understudy for Gisting Evaluation) was introduced in 2004 [44] in order to solve the drawbacks of BLEU at ISI (Information Science Institute). It is approximately based on BLEU; however, it focuses instead on recall …

Faster Transformers for Document Summarization
V Kosaraju, YD Ang, Z Nabulsi – vineetkosaraju.com
… ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a metric that measures co-occurrence statistics (7). There are numerous different flavors of ROUGE that we utilize, including ROUGE-N with N = 2 and ROUGE-L-F1 …

Idiap Abstract Text Summarization System for German Text Summarization Task
S Parida, P Motlicek – Proceedings of the 4th edition of the …, 2019 – pdfs.semanticscholar.org
… The automatic eval- uation results based on the dev and test set are shown in Table 2 with sample summaries in Ta- ble 3. To evaluate the proposed algorithms, we use ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score, which is a popular metric for text …

Towards Storytelling Automatic Textual Summerized
Y Bounab, JM Adeegbe, M Oussalah – jultika.oulu.fi
… B. Metrics: The most common metrics employed for evaluating textual summarization systems are: ROUGE This is an abbreviation for Recall-Oriented Understudy for Gisting Evaluation. Its importance is to evaluate text summarizations …

On the Effects of Automatic Transcription and Segmentation Errors in Hungarian Spoken Language Processing
MÁ Tündik, V Kaszás, G Szaszák – Periodica Polytechnica Electrical …, 2019 – pp.bme.hu
… 254–262, 2019|257 3.2 Measuring document similarities with ROUGE The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric family [26] is commonly used in text summarization, that is why we selected it for our docu- ment similarity related experiments …

High?quality tweet generation for online behavior security management based on semantics measurement
T Qin, B Wang, Z Liu, Z Chen… – Transactions on Emerging …, 2019 – Wiley Online Library
Abstract Behavior security management refers to monitoring and guiding the user’s opinions in online social networks to reduce their harmful influence to social public security. Pushing designed tw…

Summarizing Legal Rulings: Comparative Experiments
D de Vargas Feijo, VP Moreira – acl-bg.org
… On average, the ementa has a little less than 10% of the size of the source input. 4.2 Official Rouge Script The standard evaluation metric for text summa- rization is called Recall-Oriented Understudy for Gisting Evaluation (ROUGE) …

Summarizing Legal Rulings: Comparative Experiments
D Feijo, V Moreira – Proceedings of the International Conference on …, 2019 – aclweb.org
… On average, the ementa has a little less than 10% of the size of the source input. 4.2 Official Rouge Script The standard evaluation metric for text summa- rization is called Recall-Oriented Understudy for Gisting Evaluation (ROUGE) …

Automatic Text Summarization of Legal Cases: A Hybrid Approach
V Pandya – arXiv preprint arXiv:1908.09119, 2019 – arxiv.org
… 3]. ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation and it counts the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans …

A Text Abstraction Summary Model Based on BERT Word Embedding and Reinforcement Learning
Q Wang, P Liu, Z Zhu, H Yin, Q Zhang, L Zhang – Applied Sciences, 2019 – mdpi.com
… To verify the performance of the model, we compare it with the current popular automatic text summary model on the CNN/Daily Mail dataset, and use the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metrics as the evaluation method …

A Three-Stage Training Method for Abstractive Summarization
L Shi, D Liu, G Liu – ajiips.com.au
… 6.3 Evaluation Metric ROUGE [10] is adopted to measure the performance of models. ROUGE is short for Recall-Oriented Understudy for Gisting Evaluation, which is usually used to evaluate the performance of translation and summarization …

Extractive summary: an optimization approach using bat algorithm
A Pattanaik, S Sagnika, M Das, BSP Mishra – … and Computer Systems, 2019 – Springer
… Then, the efficiency of the algorithm is calculated using ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [23] package. Rouge-1 and rouge-2 are calculated. The average result of F-measure, precision and recall for all these is given in the table below …

Cross-Lingual Korean Speech-to-Text Summarization
HJ Yoon, DT Hoang, NT Nguyen, D Hwang – Asian Conference on …, 2019 – Springer
… truth. We used the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) method proposed by Lin et al. [13] to calculate the differences between the distribution of words in the prediction results and the ground truth. The …

Captioning ultrasound images automatically
AT Papageorghiou, L Drukker, P Chatelain, M Alsharid… – 2019 – ora.ox.ac.uk
… Different model configurations were compared us- ing the established general metrics BLEU (Bilingual Evaluation Understudy) and ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence), a grammar score GB (GrammarBot) [8], a …

Automatic Text Summarization: A New Hybrid Model Based on Vector Space Modelling, Fuzzy Logic and Rhetorical Structure Analysis
AB Ayed, I Biskri, JG Meunier – International Conference on Computational …, 2019 – Springer
… We used the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric. In Table 1, we report obtained ROUGE-1, ROUGE-2 and ROUGE-S metrics. Note that ROUGE-n (n = 1, 2) refers to the overlap of n-gram between generated and reference summaries …

Chinese Reviews Generation Based on HM-BiLSTM Model
J Yuan, Z Guo, G Chen, Y Sun… – Journal of Physics …, 2019 – iopscience.iop.org
… The objective evaluation metrics are mainly dependent on the corpus and algorithm. The BLEU (Bilingual Evaluation Understudy) algorithm and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) algorithm are the main method to evaluate text …

User Intention-Based Document Summarization on Heterogeneous Sentence Networks
HY Wang, JW Chang, JW Huang – International Conference on Database …, 2019 – Springer
… 3. 14. 17141. 5.2 Evaluation Metric. We use ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [8] toolkit (version 1.5.5) as our measurement. It has been widely used in DUC task for evaluating automatic summarization in Natural Language Processing …

A comprehensive survey of deep learning for image captioning
MDZ Hossain, F Sohel, MF Shiratuddin… – ACM Computing Surveys …, 2019 – dl.acm.org
Page 1. 118 A Comprehensive Survey of Deep Learning for Image Captioning MD. ZAKIR HOSSAIN, FERDOUS SOHEL, MOHD FAIRUZ SHIRATUDDIN, and HAMID LAGA, Murdoch University, Australia Generating a description of an image is called image captioning …

Generating summary sentences using Adversarially Regularized Autoencoders with conditional context
H Kong, W Kim – Expert Systems with Applications, 2019 – Elsevier
… 4.4. Evaluation. In this paper, we evaluated our models using the recall-oriented Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric (Lin, 2004) which determines the quality of summaries by comparing it to reference summaries …

Abstractive Text Summarization Using Enhanced Attention Model
RK Roul, PM Joshi, JK Sahoo – International Conference on Intelligent …, 2019 – Springer
… The entire corpus is divided into 3750 data points of training, and 625 each for validation and testing. The size of pretrained GloVe embeddings is fixed to 200. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score [24] is used for performance evaluation …

Improving the Processing of Question Answer Based Legal Documents
S CHAKRAVARTY, M MEHROTRA… – … : JURIX 2019: The …, 2019 – books.google.com
… sentence similarity for evaluation. Recall-Oriented Understudy for Gisting Evaluation (ROUGE)[13] can be used to com- pare generated sentences with the canonical forms annotated by human actors. We used the ROUGE-1 …

Combine clustering and frequent itemsets mining to enhance biomedical text summarization
O Rouane, H Belhadef, M Bouakkaz – Expert Systems with Applications, 2019 – Elsevier
… 4.1. Evaluation measures. To evaluate the performances of our system, we used a classical method for automatic evaluation of summaries called ROUGE toolkit 3 (ie, Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) …

A Fuzzy Approach for Sentences Relevance Assessment in Multi-document Summarization
JA Olivas, FP Romero – … Conference on Soft Computing Models in …, 2019 – books.google.com
… Summarization task. The results were measured through the ROUGE metrics (Recall-Oriented Understudy for Gisting Evaluation)[11] and compared with those obtained by the participant systems in MultiLing 2015. Page 79. A …

A Hybrid Solution To Abstractive Multi-Document Summarization Using Supervised and Unsupervised Learning
G Bhagchandani, D Bodra, A Gangan… – … and Control Systems …, 2019 – ieeexplore.ieee.org
… humans do. Finally, we evaluate our results using ROUGE[1] metric which is Recall-Oriented Understudy for Gisting Evaluation and BLEU[2] metric which is short for Bilingual Evaluation Understudy. These metrics quantitatively …

Extractive summarization using semigraph (ESSg)
S Sonawane, P Kulkarni, C Deshpande, B Athawale – Evolving Systems, 2019 – Springer
… 6.1.2 Rouge score. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Chin-Yew 2004) automatic n-gram matching toolkit is used for evaluation, which was adopted by Document Understanding Conferences (DUC) for automatic summarization evaluation …

An Augmented Transformer Architecture for Natural Language Generation Tasks
H Li, AYC Wang, Y Liu, D Tang, Z Lei, W Li – arXiv preprint arXiv …, 2019 – arxiv.org
… Our analysis leverages these models on variety of machine translation automatic evaluation metrics, including the BLEU (bilingual evaluation understudy) [27] score, the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [20] score, the METEOR (Metric for Evaluation …

Captioning Ultrasound Images Automatically
M Alsharid, H Sharma, L Drukker, P Chatelain… – … Conference on Medical …, 2019 – Springer
… Different model configurations were compared using the established general metrics BLEU (Bilingual Evaluation Understudy) [16] and ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence) [11], a grammar score GB (GrammarBot) [8], a …

Summarization of news articles
O Beronius – 2019 – odr.chalmers.se
… Finally, it was discovered that the used evaluation metric, ROUGE, could not be considered complete for the given task, and it would thus be advised to explore additional evaluation metrics for summarization models … 18 4 Results & Discussion 21 4.1 Evaluation – ROUGE …

Faster Transformers for Document Summarization
DA Yap, V Kosaraju, Z Nabulsi – pdfs.semanticscholar.org
… We measure speed in training and inference time, as one of our goals of our research is to increase the efficiency of existing approaches. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a metric that measures co-occurrence statistics (18) …

SUGAMAN: describing floor plans for visually impaired by annotation learning and proximity-based grammar
S Goyal, S Bhavsar, S Patel, C Chattopadhyay… – IET Image Processing, 2019 – IET
… For that purpose several metrics for example Bilingual Evaluation Understudy (BLEU) [ 40 ], Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [ 41 ], Metric for Evaluation of Translation with Explicit Ordering (METEOR) [ 42 ] etc., have been proposed …

Deep Reinforcement Learning for Text and Speech
U Kamath, J Liu, J Whitaker – Deep Learning for NLP and Speech …, 2019 – Springer
In this chapter, we investigate deep reinforcement learning for text and speech applications. Reinforcement learning is a branch of machine learning that deals with how agents learn a set of actions…

CSMDSE-Cuckoo Search Based Multi Document Summary Extractor: Cuckoo Search Based Summary Extractor
R Rautray, RC Balabantaray, R Dash… – International Journal of …, 2019 – igi-global.com
… Performanceofeachsummarygeneratedbydifferentmodels isanalyzedintermsofsentence- sentencesimilarity,ROUGEscore,and … isprovided in Table 4. The result obtained by different summary extractor is analyzed by Recall-Oriented UnderstudyforGistingEvaluationtool. 6.1 …

A survey on evaluation of summarization methods
L Ermakova, JV Cossu, J Mothe – Information Processing & Management, 2019 – Elsevier
… Automatic summarization. Text compression. Evaluation campaigns. Assessment metrics. Extraction. Extractive summarization. ROUGE. 1. Introduction … 5. short summaries focused by questions. The summaries in tasks 1-4 were evaluated by ROUGE metric (Lin & Hovy, 2003) …

Summarization of legal judgments using gravitational search algorithm
A Kanapala, S Jannu, R Pamula – Neural Computing and Applications, 2019 – Springer
… Therefore, we consider 30% of the legal document is considered as the summary. Then, ROUGE [24] (recall-oriented understudy for gisting evaluation) tool evaluates the summery with the human-generated summary. 5.1 Data …

A Fuzzy Approach for Sentences Relevance Assessment in Multi-document Summarization
E Valladares-Valdés, A Simón-Cuevas… – … Workshop on Soft …, 2019 – Springer
… The results were measured through the ROUGE metrics (Recall-Oriented Understudy for Gisting Evaluation) [11] and compared with those obtained by the participant systems in MultiLing 2015. The rest of the paper is organized as follows: Sect …

Comprehensive Event Storyline Generation from Microblogs
W Sun, Y Wang, Y Gao, Z Li, J Sang, J Yu – Proceedings of the ACM …, 2019 – dl.acm.org
… 4.1 Evaluation metric 4.1.1 ROUGE. Recall-Oriented Understudy for Gisting Evaluation (ROUGE)[5] is an evaluation toolkit for document summarization, which is widely applied to evaluate the quality of storyline genera- tion, such as [4][11] …

Performance analysis of Modified Shuffled Frog leaping Algorithm for Multi-document Summarization Problem
R Rautray, R Dash, R Dash – Informatica, 2019 – informatica.si
… over DUC dataset and evaluated by ROUGE [23, 24]. ROUGE stands for Recall- Oriented Understudy for Gisting Evaluation. It is used as the official evaluation metric for text summarization. A set of metrics such as ROUGE-L …

Multi-Document Summarization Based on Keyword Fusion
S Alshahrani, M Bikdash – 2019 SoutheastCon, 2019 – ieeexplore.ieee.org
… B. Simulations We selected 12 documents and calculated the similarity between these documents to see the performance of each tool. We used the ROUGE similarity metric which stands for Recall-Oriented Understudy for Gisting Evaluation …

An Optimized Abstractive Text Summarization Model Using Peephole Convolutional LSTM
M Rahman, FH Siddiqui – Symmetry, 2019 – mdpi.com
… The evaluation techniques based on recall, precision, and F-measure are called intrinsic evaluation. The intrinsic evaluation techniques like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [13] and BLEU [14] are widely used for judging the quality of a summary …

Extractive Document Summarization using Non-negative Matrix Factorization
A Khurana, V Bhatnagar – … Conference on Database and Expert Systems …, 2019 – Springer
… Performance of the proposed methods is evaluated using ROUGE toolkit [19]. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is an evaluation toolkit for evaluating the performance of system (algorithmic) summary against a set of reference summaries …

A text summarization method based on fuzzy rules and applicable to automated assessment
FB Goularte, SM Nassar, R Fileto, H Saggion – Expert Systems with …, 2019 – Elsevier
… The proposed approach was compared with other methods including a naive baseline, Score, Model and Sentence, using ROUGE measures … In the evaluation, they used the ROUGE package and the results were compared to other systems …

Towards automatic extractive text summarization of A-133 Single Audit reports with machine learning
VT Chou, LA Kent, JA Góngora, S Ballerini… – arXiv preprint arXiv …, 2019 – arxiv.org
… Alternatively, automated evaluation methods are implemented by packages such as Recall-Oriented Understudy for Gisting Evaluation, or ROUGE [29] and its precursor, BLEU [30], which calculate precision and recall scores that reflect the quality of a summary …

Paraphrase Generation with Collaboration between the Forward and the Backward Decoder
Q Wang, J Ren – … Joint Conference on Neural Networks (IJCNN …, 2019 – ieeexplore.ieee.org
… a) Automatic evaluation: In this work, the automatic evaluation metrics are BiLingual Evaluation Understudy (BLEU) [27] and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [28], which are different metrics used for evaluating automatic generation systems in NLP …

A survey on contrastive opinion summarisation
SK Lavanya, B Parvathavarthini – International Journal of …, 2019 – inderscienceonline.com
… 4.2 Measures The most commonly used measures are precision and aspect coverage. Recall-oriented understudy for gisting evaluation (ROUGE) is a widely used measure for summarisation. The combination of precision and recall results in F-measure …

A Novel Approach for Ontology-Driven Information Retrieving Chatbot for Fashion Brands
A Nazir, MY Khan, T Ahmed, SI Jami… – Editorial Preface From …, 2019 – researchgate.net
… RESULTS AND EVALUATION A variety of techniques are available to evaluate system performance like BiLingual Evaluation Understudy (BLEU) [34], METEOR [35], and Recall-Oriented Understudy for Gisting Evaluation (ROUGE)[36]; but these metrics are often accounted as …

ESSMArT way to manage customer requests
M Nayebi, L Dicke, R Ittyipe, C Carlson… – Empirical Software …, 2019 – Springer
… Second, we compared the results of automated extractive summarization with the summaries created by human experts. For this purpose, we used Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004). ROUGE …

Improvement of query-based text summarization using word sense disambiguation
N Rahman, B Borah – Complex & Intelligent Systems, 2019 – Springer
Page 1. Complex & Intelligent Systems https://doi.org/10.1007/s40747-019-0115-2 ORIGINAL ARTICLE Improvement of query-based text summarization using word sense disambiguation Nazreena Rahman1 · Bhogeswar Borah2 …

Abstractive Multi-Document Text Summarization Using a Genetic Algorithm
VN Mendoza, Y Ledeneva… – Mexican Conference on …, 2019 – Springer
… similarity with the title [28]. Also, we will consider other text models like sn-grams [27] and MFS [7], and other language [24]. Footnotes. 1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) version 1.5.5. References …

AI-Powered Text Generation for Harmonious Human-Machine Interaction: Current State and Future Directions
Q Zhang, B Guo, H Wang, Y Liang, S Hao… – arXiv preprint arXiv …, 2019 – arxiv.org
… 2) ROUGE (Recall-Oriented Understudy for Gisting Evaluation) ROUGE evaluates the abstract based on the co-occurrence information of n-gram, which is an evaluation method oriented to the recall rate of n-gram words. Its …

Accelerating Clinical Trial Documentation Through Human Aided Medical Writing Automation System
S Manchanda, A Indani… – TENCON 2019-2019 IEEE …, 2019 – ieeexplore.ieee.org
… METEOR (Metric for Evaluation of Translation with Explicit ORdering), BLEU (BiLingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Metrics [2]. Currently proposed system is in development with for Single Document Summarization …

Text Summarization
C Room – Architecture, 2019 – devopedia.org
… In 2004, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) was created to automate evaluation by comparing against hand-crafted summaries. ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, and ROUGE-SU are some metrics in this family …

An efficient framework of utilizing the latent semantic analysis in text extraction
AH Ababneh, J Lu, Q Xu – International Journal of Speech Technology, 2019 – Springer
… Rouge Evolution Toolkit is an evaluation software developed by Lin (2004). It stands for Recall-Oriented Understudy for Gisting Evaluation, and it evaluates the quality of the automatically generated summaries by comparing them with human-generated summaries (called …

Text summarization from legal documents: a survey
A Kanapala, S Pal, R Pamula – Artificial Intelligence Review, 2019 – Springer
… English news. 3.4.1 Evaluation metrics. ROUGE (Lin 2004) means Recall-Oriented Understudy for Gisting Evaluation. It automatically measures the quality of human generated summary by using its measures based on n-gram co-occurrence statistics …

Text Summarization in Indian Languages: A Critical Review
N Baruah, SK Sarma… – 2019 Second International …, 2019 – ieeexplore.ieee.org
… Recall-Oriented Understudy for Gisting Evaluation (ROGUE) tool kit is also applied to an agglutinative language approach to calculate the number of repetitions … phases are performed [19].Only one text summarization approach in Indian language had used ROUGE tool kit[7] …

A Comprehensive Survey on Extractive and Abstractive Techniques for Text Summarization
A Mahajani, V Pandya, I Maria, D Sharma – Ambient Communications and …, 2019 – Springer
… testing. ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation and BLUE stands for Bilingual Evaluation Understudy. These testing paradigms focus on matching the N-Grams of the candidate and reference text …

Mobile IoT device summarizer using P2P web search engine and inherent characteristic of contents
S Park, BR Cha, K Chung, JW Kim – Peer-to-Peer Networking and …, 2019 – Springer
… summary algorithms [36]. The precision (P) measure [3, 5] and the ROUGE (recall-oriented understudy for gisting evaluation) [36] performance evaluation software are employed to mea- sure our proposed summarizer. The ROUGE …

Feature Engineering and Ensemble-based Approach for Improving Automatic Short-answer Grading Performance
A Sahu, PK Bhowmick – IEEE Transactions on Learning …, 2019 – ieeexplore.ieee.org
… This motivated us to use measures such as Recall Oriented Understudy for Gisting Evaluation (ROUGE) namely ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S and ROUGE-SU [23] that evaluate the quality of the computer generated summaries in this context …

Deep Architectures for Abstractive Text Summarization in Multiple Languages
AM Zaki, MI Khalil, HM Abbas – 2019 14th International …, 2019 – ieeexplore.ieee.org
… comparing the generated sentence to the provided reference sentence. Rouge metric stands for Recall-Oriented Understudy for Gisting Evaluation which is a testing metric that measures 7https://github.com/yaserkl/RLSeq2Seq …

Mining Summary of Short Text with Centroid Similarity Distance
N Franciscus, J Wang, B Stantic – International Conference on Advanced …, 2019 – Springer
… We run the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metrics which measures summary quality by counting overlapping units such as n-grams word sequences and word pairs between the system generated summary and the gold standard summary …

Specificity-Based Sentence Ordering for Multi-Document Extractive Risk Summarization
B Ekmekci, E Hagerman, B Howald – arXiv preprint arXiv:1909.10393, 2019 – arxiv.org
… These human summaries, also thresholded at 100 words, were used in ‘intrinsic’ comparison evalu- ations (how informative the summaries are) with ROUGE – Recall-Oriented Understudy for Gisting Evaluation (Lin, 2004) and BLEU (Bilingual Eval- uation Understudy) (Papineni …

Measuring Knowledge Gaps in Student Responses by Mining Networked Representations of Texts
C Qiao, X Hu – Proceedings of the 9th International Conference on …, 2019 – dl.acm.org
… size of response vocabulary gain after merging with assessment knowledge; size of assessment knowledge vocabulary gain after merging with response; BLEU (BiLingual Evaluation Understudy) [19] and Rouge (Recall-Oriented Understudy for Gisting Evaluation)-1, -2 and -l …

Content-Based Creative Suggestions for User Queries.
S Anand – 2019 – repository.lib.ncsu.edu
… queries). Bi-Lingual Evaluation Understudy (BLEU) [Papineni et al., 2002], Recall Oriented Understudy for Gisting Evaluation (ROUGE) [Lin, 2004], and their variants use word-overlap to compare the quality of the suggestions …

Talk2Me: Automated linguistic data collection for personal assessment
M Komeili, C Pou-Prom, D Liaqat, KC Fraser… – PloS one, 2019 – journals.plos.org
… Story recall scores. We transcribe the audio recordings of story recall, and score the task using the ROUGE score (ie, ‘recall-oriented understudy for gisting evaluation’) [52]. ROUGE is typically used to evaluate automatic summarization software, and compares a candidate …

Captioning Near-Future Activity Sequences
T Mahmud, M Billah, M Hasan… – arXiv preprint arXiv …, 2019 – arxiv.org
Page 1. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 Captioning Near-Future Activity Sequences Tahmida Mahmud, Mohammad Billah, Mahmudul Hasan, and Amit K. Roy-Chowdhury, Fellow, IEEE …

Text Summarization using Partial Textual Entailment based Graphs
M Kaur, D Srivastava – … on Machine Learning, Big Data, Cloud …, 2019 – ieeexplore.ieee.org
… C. Evaluation Metrics We have evaluated the method’s performance using ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [22] which has been widely used for evaluating summaries. ROUGE: A set of metrics developed …

Query-focused abstractive summarization using sequence-to-sequence and transformer models
MMH Polash – 2019 – opus.uleth.ca
… We evaluate the output of our models against the human-created reference summary. Our transformer model outperforms our sequence-to-sequence model in all ROUGE scores. iv Page 6 … 49 4.3 ROUGE(1, 2, L) scores of the different models on the test set …

Abstractive Text Summarization On Wikihow Dataset Using Sentence Embeddings
B TOZYILMAZ – 2019 – etd.lib.metu.edu.tr
… RNN Recurrent neural network ROUGE Recall-oriented understudy for gisting evaluation Seq2Seq Sequence to sequence xiv … 2.4.2 ROGUE Recall-oriented understudy for gisting evaluation (ROUGE) [15] is another text sim- ilarity metric widely used for summarization …

A Modularized Architecture of Multi-Branch Convolutional Neural Network for Image Captioning
S He, Y Lu – Electronics, 2019 – mdpi.com
Image captioning is a comprehensive task in computer vision (CV) and natural language processing (NLP). It can complete conversion from image to text, that is, the algorithm automatically generates corresponding descriptive text according to the input image. In this paper, we present …

A novel approach for text summarization using optimal combination of sentence scoring methods
P Verma, H Om – S?dhan?, 2019 – Springer
… 5.1 Experimental set-up. Experiments are done using the ROUGE tool [42], which evaluates a summarization system on the basis of N-gram, skip-bigram plus unigram, skip-bigram, longest common sequence and weighted longest common sequence …

Data summarization: a survey
M Ahmed – Knowledge and Information Systems, 2019 – Springer
Summarization has been proven to be a useful and effective technique supporting data analysis of large amounts of data. Knowledge discovery from data (KDD) is time consuming, and summarization is an…

Abstractive Multi-Document Text Summarization Using a Genetic Algorithm
RA García-Hernández – … MCPR 2019, Querétaro, Mexico, June 26 …, 2019 – books.google.com
… TE (Textual Entailment)[18]: It consists of using textual implication in ATS that has been considered as a useful approach for obtaining a preliminary summary, 1 ROUGE (Recall-Oriented Understudy for Gisting Evaluation) version 1.5. 5. Page 431 …

Speech Summarization for Tamil Language
A NithyaKalyani, S Jothilakshmi – Intelligent Speech Signal Processing, 2019 – Elsevier
… 7.8.1. ROUGE. Recall-Oriented Understudy for Gisting Evaluation [66] is the commonly employed evaluation metric to analyze the summarization results. It can be in various forms that are discussed below: 7.8.1.1. ROUGE-n …

Geospatial relation captioning for high-spatial-resolution images by using an attention-based neural network
J Chen, Y Han, L Wan, X Zhou… – International Journal of …, 2019 – Taylor & Francis
… based Image Description Evaluation) (Vedantam, Lawrence Zitnick, and Parikh 2015) and ROUGE_L (Recall- Oriented Understudy for Gisting Evaluation) (Flick 2004), are … “ROUGE: A Package for Automatic Evaluation of Summaries.” In Proceedings of the Workshop on Text …

A survey on Automatic Text Summarization
N Nazari, MA Mahdavi – Journal of AI and Data Mining, 2019 – jad.shahroodut.ac.ir
… Lexical chain LC will be considered as a strong chain if Utility (LC) > Score Chain (L). Evaluation of the results of this method, which is done by the Rouge tool, gives the average precision of 0.45 and the average recall of 0.54 …

Automatic summarization of earnings releases: attributes and effects on investors’ judgments
E Cardinaels, S Hollander, BJ White – Review of Accounting Studies, 2019 – Springer
… To validate these evaluations, we use a technique from the field of information retrieval, known as recall-oriented understudy for gisting evaluation (ROUGE), in which summaries are evaluated against a summary prepared by an experienced investor relations officer …

On Identifying Cited Texts for Citances and Classifying Their Discourse Facets by Classification Techniques.
JY Yeh, TY Hsu, CJ Tsai, PC Cheng… – J. Inf. Sci. Eng., 2019 – pdfs.semanticscholar.org
… ROUGE score: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [34] is employed to count as relevance the number of overlapping units, eg, n-grams, word sequences, and word pairs, between c and r. The selected ROUGE metrics include ROUGE-N (N = 1, …, 4 …

A survey on deep neural network-based image captioning
X Liu, Q Xu, N Wang – The Visual Computer, 2019 – Springer
Image captioning is a hot topic of image understanding, and it is composed of two natural parts (“look” and “language expression”) which correspond to the two most important fields of artificial…

Summarizing videos into a target language: Methodology, architectures and evaluation
K Smaïli, D Fohr, CE González-Gallardo… – Journal of Intelligent …, 2019 – content.iospress.com
… Instead, we complemented our method with the tag-based method. It is also necessary to mention about Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [29]. We used a few of its concepts including Precision, Recall and F1 score …

Hierarchical deep neural network for image captioning
Y Su, Y Li, N Xu, AA Liu – Neural Processing Letters, 2019 – Springer
… descriptions using web- scale n-grams. In: CNLL, association for computational linguistics, pp 220–228 24. Lin C (2005) Recall-oriented understudy for gisting evaluation (rouge). Retrieved 20 Aug 2005 25. Lin TY, Maire M, Belongie S …

Three-step coreference-based summarizer for Polish news texts
M Kope? – Poznan Studies in Contemporary Linguistics, 2019 – degruyter.com
AbstractThis article addresses the problem of automatic summarization of press articles in Polish. The main novelty of this research lays in the proposal of a three-step summarization algorithm which benefits from using coreference information.In related work section, all coreference …

A Short Review on Image Caption Generation with Deep Learning
S Amirian, K Rasheed, TR Taha… – Proceedings of the …, 2019 – csce.ucmss.com
… ROUGE: Recall-Oriented Understudy for Gisting Evaluation determines the quality of a summary by comparing it to other summaries created by humans. ROUGE, similar to BLEU, has different n-grams based versions [42]. SPICE: Anderson et al …

Extractive Multi-document Summarization of News Articles
H Grant – 2019 – diva-portal.org
… evaluations. More traditional precision and recall based metrics are the Bilingual Evaluation Under- study (BLEU) (Papineni et al., 2002) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004) metrics. These …

Long story short: finding health advice with informative summaries on health social media
YH Liu, X Song, SF Chen – Aslib Journal of Information Management, 2019 – emerald.com
Whether automatically generated summaries of health social media can aid users in managing their diseases appropriately is an important question. The purpose of this paper is to introduce a novel text summarization approach for acquiring the most informative summaries from …

Worst-Case Polynomial-Time Exact MAP Inference on Discrete Models with Global Dependencies
A Bauer, S Nakajima – arXiv preprint arXiv:1912.12090, 2019 – arxiv.org
Page 1. 1 Worst-Case Polynomial-Time Exact MAP Inference on Discrete Models with Global Dependencies Alexander Bauer, and Shinichi Nakajima Abstract—Considering the worst-case scenario, junction tree algorithm remains …

Joint Lifelong Topic Model and Manifold Ranking for Document Summarization
J Lin, R Liu, Q Jia – arXiv preprint arXiv:1907.03224, 2019 – arxiv.org
… for document summarization. It combines the advantage of semantic features and original features of documents. It has been proved to enhance the quality of summary with 1% to 2% on ROUGE F-measure. (2) Lifelong Topic …

A Model for Automatic Abstractive Multidocument Domain-Specific Summarization
HAME Ahmed – 2019 – repository.sustech.edu
… NLP Natural Language Processing PAS Predicate Argument Structure POS Parts-Of-Speech Tagging ROUGE Recall Oriented Understudy for Gisting Evaluation RS Rich Semantic Graph SRL Semantic Role Labeling SVM Support Vector Machine TS Text Summarization …

Arabic Multi-Objective Optimization with K-mediod Clustering for Multi-Document Summarization
R Alqaisi – 2019 – pdfs.semanticscholar.org
… AQBTSS Query-Based Text Summarization System ACBTSS Arabic Concept-Based Text Summarization System ROUGE Recall-Oriented Understudy for Gisting Evaluation LSA Longest Common Sequence jMetal Metaheuristic Algorithms in Java …

Recent Advances in NLP: The Case of Arabic
MA Elaziz, MAA Al-qaness, AA Ewees, A Dahou – Springer
… bag-of-words paragraph vectors PV-DM Distributed memory paragraph vectors QDA Quadratic discriminant analysis RBF Radial basis function RF Random forest RNN Recurrent neural network ROUGE Recall-Oriented Understudy for Gisting Evaluation SA Sentiment analysis …

Systematic literature review of fuzzy logic based text summarization
A Kumar, A Sharma – Iranian Journal of Fuzzy Systems, 2019 – ijfs.usb.ac.ir
… The Accuracy is given as per the original studies, whether they are evaluated using Precision (P), Recall (R), Fitness (F), ROUGUE -1 , ROUGE-2. Sr. No … It Performs better than Fuzzy logic and neural network. Evaluation was not done using ROUGE …

Rumour Detection on Social Media for Crisis Management
S Han, F Ciravegna – … of the 16th International Conference on …, 2019 – researchgate.net
… 2015; Nguyen et al. 2015; Rudra et al. 2015, 2016, 2018). The state-of-the-art work on event summarisation usually focuses on achieving higher Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scores (Lin 2004) …

Headline Generation as a Sequence Prediction with Conditional Random Fields
CA Colmenares, M Litvak, A Mantrach… – Multilingual Text …, 2019 – World Scientific
Page 1. Chapter 7 Headline Generation as a Sequence Prediction with Conditional Random Fields Carlos A. Colmenares ?, , Marina Litvak †,?? , Amin Mantrach ‡,†† , Fabrizio Silvestri §,‡‡ , and Horacio Rodr?guez ¶,§§ ? Google …

Entailment and Spectral Clustering based Single and Multiple Document Summarization
A Gupta, M Kaur, A Bajaj, A Khanna – International Journal of …, 2019 – mecs-press.net
… It has been computed using Semantic Informativeness (SI) defined in [25]. 1) ROUGE: It is used to evaluate performance of the methods [30]. ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation and measures …

Sound Active Attention Framework for Remote Sensing Image Captioning
X Lu, B Wang, X Zheng – IEEE Transactions on Geoscience and …, 2019 – ieeexplore.ieee.org
… Captioning The metrics adopted for remote sensing image captioning are BiLingual Evaluation Understudy (BLEU) [53], Recall- Oriented Understudy for Gisting Evaluation (ROUGE_L) [54 … 7 2) ROUGE_L: ROUGE stands for Recall-Oriented Under- study for Gisting Evaluation …

Parallelizing a multi-objective optimization approach for extractive multi-document text summarization
JM Sanchez-Gomez, MA Vega-Rodríguez… – Journal of Parallel and …, 2019 – Elsevier
Skip to main content Skip to article …

Language features in extractive summarization: Humans Vs. Machines
I Arroyo-Fernández, A Curiel… – Knowledge-Based Systems, 2019 – Elsevier
… Several well-known statistical ATS evaluation methods consider lexical overlapping as an indicator of how good a summary is. The standard approach uses the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric …

Data summarization: a survey
A Mohiuddin – Knowledge and Information Systems, 2019 – search.proquest.com
Page 1. Knowl Inf Syst (2019) 58:249–273 https://doi.org/10.1007/s10115-018-1183- 0 SURVEY PAPER Data summarization: a survey Mohiuddin Ahmed1 Received: 4 April 2017 / Revised: 5 January 2018 / Accepted: 14 March …

Abstractive Text Summarization using Peephole Convolutional LSTM
MM Rahman – 2019 – duet.ac.bd
… The evaluation techniques based on recall, precision, and F-measure are called intrinsic evaluation. The intrinsic evaluation techniques like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [17] and BLEU [18] are widely used for judging the quality of a summary. In …

A Sequence-to-Sequence Text Summarization Model with Topic Based Attention Mechanism
HX Pan, H Liu, Y Tang – … Conference on Web Information Systems and …, 2019 – Springer
… 4.2 Evaluating Indicator. For the evaluation of the text summaries, this paper uses the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) automatic testing tool, which is proposed by Lin, and it is widely used in the field of automatic text summarization …

Rule-Based Automatic Question Generation Using Semantic Role Labeling
O Keklik, T Tuglular, S Tekir – IEICE TRANSACTIONS on …, 2019 – search.ieice.org
… n-grams, METEOR uses a direct word-ordering penalty. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric, which is first proposed by Lin (2004), is designed to compare n-grams recall of machine produced translations against human-produced translations …

Question Generation with Adaptive Copying Neural Networks
X Lu – 2019 – curve.carleton.ca
… We con- firmed that our method can exceed the most advanced QG method in BLEU and ROUGE evaluation scores; it also showed better performance in human evaluation. In addition, we combined the summarization scores with our model which resulted …

Automatic Semantic Categorization of News Headlines using Ensemble Machine Learning: A Comparative Study
R Bogery, N Al Babtain, N Aslam, N Alkabour… – pdfs.semanticscholar.org
… on the TF-IDF. Their experiment was conducted on DUC2006 and DUC2007 datasets they used Recall-Oriented Understudy for Gisting Evaluation (ROUGE) to evaluate the text summary results. Both models produce good …

A graph is worth a thousand words: Telling event stories using timeline summarization graphs
J Ansah, L Liu, W Kang, S Kwashie, J Li… – The World Wide Web …, 2019 – dl.acm.org
Page 1. A Graph is Worth a Thousand Words: Telling Event Stories using Timeline Summarization Graphs Jeffery Ansah? University of South Australia jeffery.ansah@mymail. unisa.edu.au Lin Liu University of South Australia lin.liu@unisa.edu.au …

Phrase-based image caption generator with hierarchical LSTM network
YH Tan, CS Chan – Neurocomputing, 2019 – Elsevier
… results. We also extend our experiment to include the MS-COCO dataset [21], and evaluate our results on four additional evaluation metrics (ie METEOR [23], ROUGE [24], CIDEr [25] and SPICE [26]). 2. Related works. The image …

Sequential transfer learning in NLP for text summarization
P Fecht – inovex.de
… NSP Next Sentence Prediction OOV Out-of-Vocabulary PG Pointer Generator RL Reinforcement Learning RNN Recurrent Neural Network ROUGE Recall-Oriented Understudy for Gisting Evaluation RR Repetition Rate Seq2seq Sequence-to-sequence SOTA State-of-the-art …

Deep learning of graph transformations
GK Andrea, K Ádám – 2019 – hlt.bme.hu
… The ROUGE score metric has been developed in 2004 by Chin-Yew Lin [13]. ROUGE is short for Recall-Oriented Understudy for Gisting Evaluation. The metric is used to determine the quality of a generated summary by comparing it to a set of human written summaries …

Text Summarization for Chatbots
M Lustig – 2019 – support.dce.felk.cvut.cz
… 34 6.2 Metrics . . . . . 35 6.2.1 Human . . . . . 35 6.2.2 ROUGE . . . . . 35 6.2.3 BLEU . . . . . 37 6.2.4 Run Time …

Neural generation of textual summaries from knowledge base triples
P Vougiouklis – 2019 – eprints.soton.ac.uk
… ROUGE Recall-Oriented Understudy for Gisting Evaluation (Lin, 2004) (see Section 3.1.1) RDF Resource Description Framework RLM Recurrent Language Model RNN Recurrent Neural Network S3 Semantic Sentence Simplification (Mrabet et al., 2016) WD1 Wikidata truthy …

The effect of noise in the training of convolutional neural networks for text summarisation
A Meechan-Maddon – 2019 – diva-portal.org
Page 1. The effect of noise in the training of convolutional neural networks for text summarisation Master’s Thesis Ailsa Meechan-Maddon Uppsala University Department of Linguistics and Philology Master Programme in Language …

Using a Character-Based Language Model for Caption Generation
S Keisala – 2019 – diva-portal.org
Page 1. Linköpings universitet SE– Linköping + , www.liu.se Linköping University | Department of Computer and Information Science Master thesis, 30 ECTS | Datateknik 2019 | LIU-IDA/LITH-EX-A–19/095–SE Using a Character-Based Lan- guage Model for Caption Gener …

Domain Category Information as a Guide for Sentence Ranking to Support Medical Text Summarization
KE Maduabunachukwu – 2019 – atrium2.lib.uoguelph.ca
… 73 4.3 Overlap between summaries created ….. 79 4.4 Examining the ROUGE Results in Detail using Selected Documents ….. 80 … 102 4.7.1 Overall weakness of ROUGE-L scores on this data set ….. 103 …

Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments
M Fomicheva, L Specia – Computational Linguistics, 2019 – MIT Press
Create a new account. Email. Returning user. Can’t sign in? Forgot your password? Enter your email address below and we will send you the reset instructions. Email. Cancel. If the address matches an existing account you will …

Text Summarisation Using Laplacian Centrality-Based Minimum Vertex Cover
A Gupta, M Kaur – Journal of Information & Knowledge Management, 2019 – World Scientific
… (a) ROUGE: ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It measures the quality of an automatically generated summary by comparing it to a \gold standard”, typically a human generated summary …

Unsupervised learning strategies for automatic generation of personalized summaries
V Woloszyn – 2019 – lume.ufrgs.br
… 27 Figure 3.1 A summarized snapshot of “Into the Wild” lesson plan…..36 Figure 3.2 Distribution of Rouge results…..39 Figure …

A Study on Neural Network Modeling Techniques for Automatic Document Summarization
CI Tsai – 2019 – diva-portal.org
Page 1. IT 19 023 Examensarbete 30 hp Juni 2019 A Study on Neural Network Modeling Techniques for Automatic Document Summarization Chun-I Tsai Institutionen för informationsteknologi Department of Information Technology Page 2. Page 3 …

Query-Focused Abstractive Summarization using Neural Networks
C Aryal – 2019 – 142.66.3.39
… 6 2.2.1 Recall-Oriented Understudy for Gisting Evaluation (ROUGE) … recall. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and Bilingual Evalu- ation Understudy (BLEU) are the two tools that we have used to evaluate the performance of our models …

Solving submodular text processing problems using influence graphs
A Vardasbi, H Faili, M Asadpour – Social Network Analysis and Mining, 2019 – Springer
Submodular functions appear in a considerable number of important natural language processing problems such as text summarization and dataset selection. Current graph-based approaches to solving such…

Machine Comprehension-Incorporated Relevance Matching
C Zhang, H Wang, L Zhou, Y Wang… – 2019 IEEE International …, 2019 – ieeexplore.ieee.org
Page 1. Machine Comprehension-Incorporated Relevance Matching Chen Zhang 2,? , Hao Wang 1,? , Liang Zhou 2,? , Yijun Wang 3 , Can Chen 3 1 Alibaba AI Labs, Beijing, China 2 360 Search Lab, Beijing, China 3 LineZone …

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model
W Cui, F Wang, X He, D Zhang, X Xu, M Yao, Z Wang… – Remote Sensing, 2019 – mdpi.com
A comprehensive interpretation of remote sensing images involves not only remote sensing object recognition but also the recognition of spatial relations between objects. Especially in the case of different objects with the same spectrum, the spatial relationship can help interpret …

Enhancing unsupervised neural networks based text summarization with word embedding and ensemble learning
N Alami, M Meknassi, N En-nahnahi – Expert systems with applications, 2019 – Elsevier
Skip to main content …

Automatic text summarization: What has been done and what has to be done
A Aries, WK Hidouci – arXiv preprint arXiv:1904.00688, 2019 – arxiv.org
Page 1. Automatic text summarization: What has been done and what has to be done Abdelkrime Aries Djamel eddine Zegour Walid Khaled Hidouci Ecole nationale Supérieure d’Informatique (ESI, ex. INI), Algiers, Algeria Emails: {ab aries, d zegour, w hidouci}@esi.dz …

Forced Attention for Image Captioning
H Devarapalli – 2019 – hammer.figshare.com
… Images and their respective descriptions from Virtual Tourist website were used for training the linear regression model. Finally, Aker and Gaizauskas used ROUGE (Lin, 2004) and manual readability as metrics for evaluating their results. Their model …

Multimodal Summarization and Beyond
A Khullar – 2019 – amankhullar.github.io
… Learning Era 8 2.1.3.1 Recurrent Neural Networks 9 2.1.3.2 Long Short Term Memory 11 2.1.3.3 Encoder-Decoder Architecture with Attention 12 2.2 Task Definition 15 2.2.1 Problem Formulation 15 2.2.2 Evaluation 15 2.2.2.1 Recall and Precision 15 2.2.2.2 ROUGE 16 2.3 …

AAEE–Automated evaluation of students’ essays in Arabic language
AM Azmi, MF Al-Jouie, M Hussain – Information Processing & Management, 2019 – Elsevier
… assessment. Then use the summaries instead of full essays to group like-quality essays. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004), is a widely used metric to evaluate the summaries. ROUGE uses …

Deep learning for image captioning: an encoder-decoder architecture with soft attention
M Gómez Martínez – openaccess.uoc.edu
Page 1. Universitat Oberta de Catalunya (UOC) Máster Universitario en Ciencia de Datos (Data Science) TRABAJO FINAL DE MÁSTER Área: Big Data / Machine Learning Deep Learning for Image Captioning An Encoder-Decoder Architecture with Soft Attention …

Probability and Semantic Models in News Summarization
A Schatz – alexschatz.us
… To test this, they use a system called ROUGE, or Recall-Oriented Understudy for Gisting Evaluation (Lin 2004). ROUGE requires a large gold-standard database of summaries in the same format as my program’s output, so it would be difficult to implement without manually …

Neural machine reading comprehension: Methods and trends
S Liu, X Zhang, S Zhang, H Wang, W Zhang – Applied Sciences, 2019 – mdpi.com
Machine reading comprehension (MRC), which requires a machine to answer questions based on a given context, has attracted increasing attention with the incorporation of various deep-learning techniques over the past few years. Although research on MRC based on deep …

Sumarizace textu pro chatboty
L Matyáš – 2019 – dspace.cvut.cz
… 34 6.2 Metrics . . . . . 35 6.2.1 Human . . . . . 35 6.2.2 ROUGE . . . . . 35 6.2.3 BLEU . . . . . 37 6.2.4 Run Time …

Transfer Hierarchical Attention Network for Generative Dialog System
X Zhang, Q Yang – International Journal of Automation and Computing, 2019 – Springer
Page 1. Transfer Hierarchical Attention Network for Generative Dialog System Xiang Zhang Qiang Yang Computer Science and Engineering Department, Hong Kong University of Science and Technology, Hong Kong, China …

Trends in integration of vision and language research: A survey of tasks, datasets, and methods
A Mogadala, M Kalimuthu, D Klakow – arXiv preprint arXiv:1907.09358, 2019 – arxiv.org
Page 1. Trends in Integration of Vision and Language Research Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods Aditya Mogadala amogadala@lsv.uni-saarland.de Marimuthu Kalimuthu mkalimuthu@lsv.uni-saarland.de …

Automatic summarisation: 25 years On
C Or?san – Natural Language Engineering, 2019 – cambridge.org
… The proliferation of machine learning and of ROUGE (Recall-Oriented Understudy for Gisting Evaluation), the most used automatic evaluation metric in automatic summarisation (see Section 4 for more details), also meant that ‘literature is turning into a giant leaderboard, where …

[BOOK][B] From extractive to abstractive summarization: A journey
P Mehta, P Majumder – 2019 – Springer
… We build upon this argument to construct several content-based aggregation techniques, achieving a substantial improvement in the ROUGE scores … 29 3.3.2 BLEU ….. 30 3.3.3 ROUGE Measure ….. 31 3.3.4 Pyramid Score ….. 31 …

Ctras: Crowdsourced test report aggregation and summarization
R Hao, Y Feng, JA Jones, Y Li… – 2019 IEEE/ACM 41st …, 2019 – ieeexplore.ieee.org
Page 1. CTRAS: Crowdsourced Test Report Aggregation and Summarization Rui Hao 1 , Yang Feng 2? , James A. Jones 2? , Yuying Li 1 , Zhenyu Chen 1? 1 State Key Laboratory for Novel Software Technology Nanjing University …

Making effective use of healthcare data using data-to-text technology
S Pauws, A Gatt, E Krahmer, E Reiter – Data Science for Healthcare, 2019 – Springer
Page 1. Making Effective Use of Healthcare Data Using Data-to-Text Technology Steffen Pauws, Albert Gatt, Emiel Krahmer, and Ehud Reiter 1 Introduction Note-taking in medicine stems back from the time of Hippocrates in …

Towards sentiment augmented predictive techniques in natural language
B Varshit – 2019 – web2py.iiit.ac.in
… 38 6.2.2 CIDEr . . . . . 38 6.2.3 METEOR . . . . . 38 6.2.4 ROUGE . . . . . 38 6.3 Dataset . . . . . 38 6.4 Sentiment …

Learning to predict text quality using Generative Adversarial Networks
W Sheriff – 2019 – diva-portal.org
… 31 2.7 Evaluation Methods . . . . . 33 2.7.1 Recall-Oriented Understudy for Gisting Evaluatiion (ROUGE) . . . . 34 2.7.2 Limitations of ROUGE . . . . . 35 2.7.3 QualitativeEvaluation …

Unsupervised summarization of public talk radio
S O’Brien – 2019 – dspace.mit.edu
… I use this evaluation dataset to show that my model quantitatively outper- forms strong rule- and graph-based unsupervised baselines on ROUGE and METEOR while qualitatively demonstrating fluency and information retention according to hu- man judges …

Unsupervised Multi-Document Summarization
M Eisele – 2019 – inf.uni-hamburg.de
Page 1. Unsupervised Multi-Document Summarization A thesis for the academic degree ‘Master of Science’ of Michael Eisele 7eisele@informatik.uni-hamburg.de Field of study: Computer Science Matriculation number: 7089760 Examiners: Prof …

Generative Adversarial Networks in Text Generation
Z Wang – 2019 – diva-portal.org
… Also, the state-of-art evaluation of the summaries is based on standard tests, including BLEU [12], ROUGE [13] and other tests which use the exact word matching as the basic principle. However, it may be problematic when coming across rephrased summaries …