A number of businesses have left downtown over the past few years or are considering leaving. Some business owners are saying higher rents are hitting newer businesses harder than more established ones that might own the building they occupy or have maintained a long-term lease.