For the first 18 years of my life I learned the true meaning of being left out, excluded from the wider society of education, social interaction and discovery. Then I left segregated education and hospital and went to a mainstream college. Finally, I had joined the real world where my life could begin. Millions of disabled people just like me experienced the same.

Now, we are in danger of being left out again if the UK votes to leave the European Union on 23 June. A “go-it-alone” Britain is one that will turn back the clock for disabled people. Weaker protection from discrimination will keep us out of the workplace. We will be grounded by watered-down rules on transport accessibility. Many products and services will be off-limits.

This is because go-it-alone Britain views the promotion and protection of human and civil rights as un-British. It believes workers’ rights are a regulatory burden. Laws and standards concerning accessibility are cast as irritating red tape. It is a Britain that would regard the equal rights of disabled people not as an aspiration, but as a problem.

Where Britain once inspired the development of disability rights across the EU, over time it has become more a follower. Our Disability Discrimination Act arrived before EU law required such measures, but it did not apply to most employers – only to those with more than 15 employees. It was as a result of the EU employment equality directive of 2000 that all employers finally came within the scope of Britain’s disability discrimination law. The rights of Britain’s six million unpaid carers to protection from workplace discrimination was only recognised following a judgment by the European court of justice in 2008. EU rules also require transport providers to provide assistance from trained staff to disabled passengers travelling by air, rail, bus and coach, and ships. It’s hard enough securing these rights now. It won’t be better without EU backing.

The current proposal for an EU directive on public-sector website accessibility will help address the exclusion of disabled people from the online world. The EU Accessibility Act will ensure that manufacturers and suppliers of a wide range of products including computers, phones, ATM and ticketing machines comply with agreed accessibility standards.

Taking account of disabled people’s history, how confident can we be that a Britain going it alone would unilaterally implement measures that would improve disabled peoples lives in the UK and protect them from discrimination? We can’t. Nigel Farage has argued that Britain’s anti-discrimination law should be scrapped altogether.

We are already witness to the pain of austerity, which has done great harm to the wellbeing, independence and opportunities of disabled people and their families. The Institute of Fiscal Studies has said that the economic impact of leaving the EU would force ministers to extend austerity for a further two years.

The choice for disabled people is a stark one: vote remain for continued progress towards becoming equal citizens of the EU and the wider world, or vote for Britain to go it alone and face the consequences of being left out in the cold again.

Jane Campbell is an active independent crossbench peer, currently working on the welfare reform bill, equality bill and coroner’s and justice bill. She convened Not Dead Yet UK, and co-ordinates the Resistance campaign