[ with exactly one argument before ] tests if the argument is a non-empty string. In your case what $? expands to is a non-empty string and the test succeeds.
[ "$?" -eq 0 ] would check if $? is numerically equal to 0. I guess this is closer to what you want to do.
Note [ is not syntax, it is a command. In general the whole thing between if and then is a snippet of shell code whose final exit status matters for if. In case of if [ "$?" -eq 0 ]; then the relevant exit status comes from [. This exit status trivially depends on the exit status of the previous command (stored in $?); why not to use the previous command directly then? You can (and should) do if echo a | grep b; then instead of playing with $?. Testing the value of $? when you can directly rely on the exit status is bad practice.
Fixed:
if echo a | grep b; then echo bfi