おねえさん and おねえちゃん both mean ‘older sister’. The difference is おねえさん sounds more formal than おねえちゃん.
For example, おねえちゃん is often used when you call your own older sister as you don’t have to be polite or formal in your own family.
You should use おねえさん even in your family if it is really strict and formal.
You use おねえさん when you refer to somebody else’s older sister.