If it seems like the same Wall Street banks keep getting mentioned again and again as examples of companies requiring full-time, in-office work, there may be a reason: They don’t have much company.

A new survey of human resources leaders from the Conference Board, a nonprofit business research group, finds that just 4% said they are requiring all employees to return to the workplace full-time. And less than half (45%) said they were requiring some workers to return to the office five days a week.