Dataset and Evaluation


Follow us on Twitter

Training Corpus

To develop your multilingual author profiling software, we provide you a training corpus, which consists of multilingual (Roman Urdu and English) SMS based author profiles (or documents). For gender, a multilingual author profile may belong to either Male or Female class. With regard to age, a multilingual author profile may fall into one of the three categories: 15-19, 20-24, 25-xx.
Gender and age information is associated with each multilingual author profile (see truth file: truth.txt, which is provided with the training corpus). All author profiles in the training corpus are stored in the .txt format to make the task easier.

Test Corpus

Once you finish tuning your multilingual author profiling software to achieve the satisfying performance on the training corpus, you should run your software on the test corpus and submit the output of your software as described below.

Performance Measure

The performance of your multilingual author profiling system will be measured using Accuracy evaluation measure.

Submission of Software and Output

You need to submit the following:
  1. Multilingual Author Profiling Software
    1. We ask you to prepare your software so that it can be executed generically for both age and gender. To maximize the sustainability of software submissions for this task, we encourage you to prepare your software so it can be re-trained on demand, i.e., by offering two commands, one for training, and one for testing. This way, your software can be reused on future evaluation corpora.
    2. The training command shall take as input, i) an absolute path to a training corpus, and ii) an absolute path to an empty output directory. Based on the training corpus, your software shall train a classification model, and save the trained model to the specified output directory (optional).
    3. The testing command shall take as input, i) an absolute path to a test corpus ii) an absolute path to a previously trained classification model (optional), and iii) an absolute path to an empty output directory. Based on the classification model, the software shall classify each case found in the test corpus and write an output in csv file(s).
    4. When your software is submission-ready, please mail your software along with output (csv) files. Email: maponsms@gmail.com
    5. Note: By submitting your software you retain full copyrights. You agree to grant us usage rights only for the purpose of the FIRE'18-MAPonSMS competition. We agree not to share your software with a third party or use it for other purposes than the FIRE'18-MAPonSMS competition.
  2. Output Files
    You can use your approach developed on the training corpus, to get predictions on the test corpus. Your output submission should be two CSV files in the following format:

  3. Gender Prediction
    Test_Author_Profile_IdGender
    Test-Document-001male
    Test-Document-002female
    Test-Document-003male
    Test-Document-004male
    Test-Document-005female
        
    Age Prediction
    Test_Author_Profile_IdAge
    Test-Document-00115-19
    Test-Document-00220-24
    Test-Document-00325-xx
    Test-Document-00415-19
    Test-Document-00525-xx

    The sample submission files for Gender and Age can be downloaded by clicking the buttons below.

  4. Working Notes
    All the participants have to email their working note paper describing their approach used to multilingual author profiling on SMS messages. Email: maponsms@gmail.com

Results

Total 19 teams registered with us for the task, while 9 submitted their systems. The following table lists the performances achieved by the participating teams:

   
Multilingual Author Profiling on SMS Performance (Accuracy)
ParticipantsGenderAgeJoint
Sharmila Devi, Subramanian Kannimuthu, G. Safeeq, and Anand Kumar
Karpagam College of Engineering, India.
0.870.650.57
D. Thenmozhi and Chandrabose Aravindan
Sri Sivasubramaniya Nadar College of Engineering, India.
0.850.630.52
Ali Nemati
The University of Washington Tacoma, USA.
0.830.600.49
Deepanshu Gaur
Maharaja Agrasen Institute of Technology (IPU), India.
0.750.640.47
Dijana Kosmajac and Vlado Keselj
Dalhousie University, Canada.
0.740.590.43
Òscar Garibo
Optical Tech & Support, Spain.
0.770.570.43
Ramsha Imran* and Muntaha Iqbal°
*CUI, Lahore campus, Pakistan and °Virtual University Lahore, Pakistan.
0.730.530.38
Asmara Safdar, Osama Akhter, Osama Inayat, and Naeem Hassan
CUI, Lahore campus, Pakistan.
0.690.530.35
Baseline0.600.510.32
Abdul Sittar and Iqra Ameer
CUI, Lahore campus, Pakistan.
0.550.370.23


A more detailed analysis of the detection performances can be found in the overview paper accompanying this task. (available on 01-10-2018)