Carnegie House's landowners want to raise the annual ground rent from $1.1 million to $6 million.