Now, Later, Never:
A Study of Urgency in Mobile Push-notifications

Beatriz Esteves, Kieran Fraser, Shridhar Kulkarni, Owen Conlan, Víctor Rodríguez-Doncel

Paper Overview

Push-notifications, by design, attempt to grab the attention of subscribers and impart new or valuable information in a particular context. These nudges are commonly initiated by marketing teams or autonomously via fixed rule sets and therefore, subsequent delivery interruptions tend to conflict with subscriber priorities and activities. In this work, we present an ontology used to aid in the annotation of urgency within a notification, based on its text content. We also demonstrate a variety of models capable of distinguishing multiple levels of urgency in a notification which could be used to help subscribers better prioritize information pushed at them, aid marketers creating campaigns and facilitate improved transparency with respect to the delivery-time chosen for pushed notifications.

Springer chapter Preprint

Additional resources

  1. Push-notification and its components
  2. Crowdsourced Annotation
  3. App Push Statistics
  4. Text Features Statistics
  5. Classification Algorithms
  6. Experiment 1 - Baseline
  7. Experiment 2 - Data Augmentation
  8. Experiment 3 - Time Expressions
  9. Experiment 4 - Delivery Date

Push-notification and its components

Key features of a common mobile push-notification:

Crowdsourced Annotation

App Push Statistics

Text Features Statistics

The advertools, TextBlob and Codeq-NLP Python packages were used to engineer a number of text features which were shown to be statistically significantly different across varying app category types, as is illustrated in the table below:

Feature χ2 p
count_stopwords 31526.06 <0.01
count_emojis 56577.36 <0.01
count_capital_words 16594.54 <0.01
count_characters 26342.83 <0.01
avg_word_length 18291.99 <0.01
count_words 22336.36 <0.01
count_numeric_chars 17007.64 <0.01

Classification Algorithms

Related research has shown that the following Machine Learning algorithms have worked well in classification tasks: The annotated dataset was split into train (80%) and test (20%) sets and two problem transformation approaches were applied for facilitating multi-label classification:
  • Classifier Chains: for every urgency label, a classifier was created and ordered as a chain such that the first classifier ingested only input features and the subsequent classifiers ingested the input features and outputs of the previous classifiers in turn. More information
  • Binary Relevance: for every urgency label, a single binary-classifier was created. The final output was the union of predictions made by each individual classifier.

Experiment 1 - Baseline

Experiment 2 - Data Augmentation

Experiment 3 - Time Expressions

List of works used to extract time expressions from the notification text:
  • SemEval-2013 Task 1 - TempEval-3: UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., & Pustejovsky, J. (2013). Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). p. 1-9. URL:
  • SemEval-2007 Task 15: Verhagen, M., Gaizauskas, R., Schilder, F., Hepple, M., Katz, G., & Pustejovsky, J. (2007). TempEval Temporal Relation Identification. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007). p. 75-80. URL:
  • SemEval-2010 Task 13: TempEval-2: Verhagen, M., Sauri, R., Caselli, T., & Pustejovsky, J. (2010). SemEval-2010 Task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation. p. 57-62. URL:
  • SemEval-2018 Task 6: Laparra, E., Xu, D., Elsayed, A., Bethard, S., & Palmer, M. (2018). Parsing Time Normalizations. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018). p. 88-96. URL:
  • SemEval-2021 Task 10: Laparra, E., Su, X., Zhao, Y., Uzuner, O., Miller, T., & Bethard, S. (2021). Source-free domain adaptation for semantic processing. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). p. 348-356. URL:
The figure below shows the performance improvement of the urgency classification algorithms when time-expression information was included as an input feature.
The table below illustrates the 10 most frequent time-expressions identified.

Time-expression Label Num. Notifications
B-Calendar-Interval 2148
B-This 1288
B-Period 830
B-Number 658
B-Frequency 574
B-After 440
B-Year 366
B-Part-Of-Day 322
B-Season-Of-Year 287
B-Last 267

Experiment 4 - Delivery Date

Get in touch

Feel free to reach out to us regarding the research presented.