The function performs Kappas for interrater agreement between two-raters, the author of the thesis and one other coder.
The calculation is based on the kappa2 function from the irr package (Gamer et al, 2012).
All scripts to use this function to perform the calculation for each interrater agreement trial in Rajeg (2019, Chapter 3) are shown in the Examples section below.
kappa_tidy( df = NULL, var_names = "pattern_(coder|author)", split_by = "synonyms", round = "pre_disc" )
| df | The input data frame for each interrater trial tasks |
|---|---|
| var_names | character strings of regular expressions for the columns involved in the calculation of the Kappa score (i.e. the interrater agreement score). |
| split_by | by default it is specified by "synonyms" since Kappas are computed for classification for data per synonyms included in the trial. |
| round | character strings indicating whether the Kappas are for the "pre-discussion" or "post-discussion" round. |
A tibble/data frame
Gamer, M., Lemon, J., & Singh, P. (2012). irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.84.
Rajeg, G. P. W. (2019). Metaphorical profiles and near-synonyms: A corpus-based study of Indonesian words for HAPPINESS (PhD Thesis). Monash University. Melbourne, Australia. https://doi.org/10.26180/5cac231a97fb1.
# kappa calculation for the constructional patterns ## pre-discussion irr_cxn <- kappa_tidy(df = df_cxn_pattern, var_names = "^pattern", round = "pre_disc") irr_cxn # check the output format#> # A tibble: 3 x 5 #> words rater cases kappa round #> <chr> <int> <int> <dbl> <chr> #> 1 kebahagiaan 2 100 0.848 pre_disc #> 2 kegembiraan 2 100 0.858 pre_disc #> 3 kesenangan 2 100 0.908 pre_disc#> [1] 0.8713882#> [1] 0.03242837# kappa calculation for the metaphoricity ## pre-discussion irr_use_1 <- kappa_tidy(df = df_meta_use_1st, var_names = "^use_(coder|author)$", round = "pre_disc") mean(irr_use_1[["kappa"]]) # mean#> [1] 0.5057692#> [1] 0.1363633## post-discussion irr_use_2 <- kappa_tidy(df = df_meta_use_2nd, var_names = "^use_2nd_", round = "post_disc") mean(irr_use_2[["kappa"]]) # mean#> [1] 0.7044434#> [1] 0.06502381# interrater agreement for the conceptual metaphors ## pre-discussion irr_cm_1 <- kappa_tidy(df = df_cm, var_names = "_pre$", round = "pre_disc") mean(irr_cm_1[["kappa"]]) # mean#> [1] 0.5157949#> [1] 0.04405836## post-discussion irr_cm_2 <- kappa_tidy(df = df_cm, var_names = "_post$", round = "post_disc") mean(irr_cm_2[["kappa"]]) # mean#> [1] 0.8618376#> [1] 0.05186339