Accuracy Metrics

Imports

import pandas as pd
from os import listdir, getcwd
from os.path import isfile, join

Data import

This notebook can be used to compute and average results across multiple runs, simply adapt the run_folders.

run_folders = ['run_1']

dfs = []
for folder in run_folders:
    run_folder = folder + "/validation/"
    file_list = [f for f in listdir(run_folder) if isfile(join(run_folder, f))]
    print("Loaded", file_list, "from", run_folder)
    for i in range(len(file_list)):
        fname = run_folder + file_list[i]
        df_temp = pd.read_csv(fname)
        df_temp["src"] = file_list[i].replace(".csv", "")
        df_temp["run"] = folder
        dfs.append(df_temp)
df = pd.concat(dfs)

Loaded ['ActiveFL.csv', 'CEP.csv', 'Random.csv', 'FedCS.csv', 'PowD.csv'] from run_1/validation/

Preview of the imported data

df.sample(5)

	round	client	loss	acc	total	correct	class_accuracy	src	run
12199	24	humongous-rating	0.031195	0.725490	51	37	[0.8571428656578064, 1.0, 0.5, 0.4000000059604...	Random	run_1
4852	9	seething-tambourine	0.056867	0.450980	51	23	[0.3333333432674408, 1.0, 1.0, 0.3333333432674...	CEP	run_1
9348	18	square-lien	0.042600	0.568627	51	29	[0.8333333134651184, 0.875, 1.0, 0.0, 0.333333...	Random	run_1
8055	16	kinetic-kern	0.035176	0.745098	51	38	[0.6666666865348816, 1.0, 0.800000011920929, 0...	PowD	run_1
3443	6	few-body	0.052351	0.509804	51	26	[0.75, 0.8333333134651184, 0.6666666865348816,...	FedCS	run_1

Generate Mean, Min and Max Accuracy after the final round

Aggregation per Run

df_gen = df[df['round'] == max(df['round'])][['src', 'run', 'acc']]
df_gen_max = df_gen.groupby(['src', 'run']).max()
df_gen_max['op'] = 'Top-1'
df_gen_mean = df_gen.groupby(['src', 'run']).mean()
df_gen_mean['op'] = 'Mean'
df_gen_min = df_gen.groupby(['src', 'run']).min()
df_gen_min['op'] = 'Bottom-1'
df_gen = pd.concat([df_gen_max, df_gen_mean, df_gen_min])
df_gen = df_gen.pivot(columns="op")
df_gen

		acc
	op	Bottom-1	Mean	Top-1
src	run
ActiveFL	run_1	0.568627	0.735216	0.901961
CEP	run_1	0.588235	0.743725	0.901961
FedCS	run_1	0.549020	0.740392	0.921569
PowD	run_1	0.568627	0.731216	0.882353
Random	run_1	0.568627	0.740000	0.901961

Final Aggregation

df_output = df_gen.groupby('src').mean()
df_output

	acc
op	Bottom-1	Mean	Top-1
src
ActiveFL	0.568627	0.735216	0.901961
CEP	0.588235	0.743725	0.901961
FedCS	0.549020	0.740392	0.921569
PowD	0.568627	0.731216	0.882353
Random	0.568627	0.740000	0.901961

Time to Accuracy

Generate Time to Accuracy (Mean)

df_gen = df[['round', 'src', 'run', 'acc']]
accuracy_steps = [0.2,0.3,0.4,0.5,0.6,0.7,0.8]
df_res = []
for algorithm in df['src'].unique():
    result = {}
    for step in accuracy_steps:
        df_tmp = df_gen[df_gen['src'] == algorithm][['round', 'acc', 'run']].groupby(['round', 'run']).mean().reset_index()
        df_tmp = df_tmp[df_tmp['acc'] >= step]
        if len(df_tmp) == 0:
            result[step] = '-'
        else:
            df_temp = df_tmp[['round', 'run']].groupby('run').min().reset_index()
            result[step] = df_temp['round'].mean()
    df_res.append(pd.DataFrame(result,index=[algorithm]))
df_res = pd.concat(df_res)
df_res.columns = ["{:.2f}".format(x) for x in df_res.columns]
df_res

	0.20	0.30	0.40	0.50	0.60	0.70	0.80
ActiveFL	4.0	5.0	7.0	9.0	13.0	25.0	-
CEP	4.0	5.0	7.0	9.0	14.0	23.0	-
Random	3.0	4.0	6.0	7.0	12.0	22.0	-
FedCS	3.0	5.0	6.0	9.0	13.0	23.0	-
PowD	4.0	4.0	6.0	8.0	14.0	24.0	-