Unable to run bpf program as non root

I am trying to run a simple bpf program that I wrote. But I am not able to run it as non root user. Below is the program I am trying to load, It basically gets the pointer to my map whose fd is map_fd (I am not showing the code where I create the map). It works as root but for some reason fails with non root user.

Output of uname -a

Linux haven 5.8.0-38-generic #43~20.04.1-Ubuntu SMP Tue Jan 12 16:39:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

BPF program

BPF_MOV64_IMM(BPF_REG_0, 0),BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4),BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),BPF_LD_MAP_FD(BPF_REG_1,map_fd),BPF_RAW_INSN(BPF_CALL | BPF_JMP, 0, 0, 0, BPF_FUNC_map_lookup_elem),BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),BPF_EXIT_INSN(),BPF_EXIT_INSN(),