In the mid-1800s, New York City was grappling with how to manage one of the most disruptive technologies it had seen to date: bicycles. The explosive popularity of urban cycling threw New York's policymakers into a tailspin, leading to cycling bans and restrictive permits before the vehicles were eventually legalized statewide.

Local governments showed a similar reception to the abrupt introduction of e-scooter services nearly 200 years later, in 2018. Some cities skeptical of the technology imposed bans or took legal action against scooter operators, while cities that were open to the concept developed pilots to assess impact, often moving on to adopt informed e-scooter programs.

While public sector pilots are not new, their use has grown rapidly over the last 20 years with the adoption of smartphones, the rise of the sharing economy and the "Cambrian explosion" of new technologies, said David Zipper, visiting fellow at the Harvard Kennedy School's Taubman Center for State and Local Governments. He explained that pilots are "a path where local leaders can navigate uncertainty" before jumping prematurely into tech adoption — or prohibition.

Few guidelines have been offered, however, in how to best create a pilot, from developing hypotheses and metrics to collecting data and sharing information with the public.

"This whole concept of pilots and how to launch a new technology in a limited way has been a bit of a mess," Zipper said. This is particularly pertinent in the mobility space, where advancements have created opportunities for cities to test everything from autonomous shuttles to shared mopeds.

In a recent policy brief for the Taubman Center, Zipper detailed the most critical steps of developing and conducting a mobility pilot to aid local governments in informing decisions around technology procurement. Smart Cities Dive caught up with Zipper to discuss his research and debunk some common misconceptions around what it means to find "success" in a pilot.

SMART CITIES DIVE: What are the variations of micromobility pilots?

DAVID ZIPPER: To correct a word, it’s really "mobility" pilots, not just micromobility. Micromobility is just scooters and bikes, but I think this applies also to microtransit or autonomous shuttles or, you could argue, sidewalk drones. Each of those technologies … there's different potential social goals that could come from them. 

Many cities have argued that scooters can be useful and that they can reduce car trips. So that's a hypothesis to test, and to their credit, many cities have tested that particular hypothesis with their scooter pilots. 

Microtransit is a good example of a technology where … I hear people talk about cost savings. It's a cheap way to move people. If that's what a transit agency thinks, I'd say test it and do a pilot. See if it's true that if we replace full-size bus service with on-demand shuttle service, our cost-per-rider is lower. That's a good pilot. 

You might also suggest that if we have sidewalk drones in our city, they would reduce the amount of double parking by UPS, DHL and so on. A worthy policy goal, but you need to measure it.

These are all theoretically appropriate questions to pose with a pilot, but you need to figure out what those questions are, set goals and be clear about those goals before you actually have the vehicles hit the streets.

Are there different approaches you need to take dependent on the mode?

ZIPPER: There's different societal goals or policy goals those technologies could achieve, so you’re testing different things with the pilot. The process of doing a pilot would be relatively the same, in terms of saying what do we hope this technology might do at scale, how can we test to see if that’s true and then how do we collect data and verify the data.

Your brief implied that, without a guideline for deploying pilots, governments could pursue pilots with ill intentions or political motivations. Can you elaborate on that?

ZIPPER: When you look at the announcement of autonomous shuttles, particularly Fairfax County and there was another one in Pinellas County, FL recently. Those are two examples of announcements that talk about how this is the first time this technology has been showcased in the Washington, DC region or in Northwestern Florida. 

That’s fine, that's good; the press loves that stuff. But if I'm a commuter I may ask, how does this relate to me? What are you actually learning from this very small-scale deployment that will ultimately lead to me having a better commute or my city being better? It's not clear.

I would suggest that many times, the reasons that pilots happen have to do with what I would call FOMO [fear of missing out]. You have a city leader who wants to throw a technology on the street before peers do. I think it's a really bad habit and it can lead to wasted resources, dashed expectations and, in an absolutely worse case scenario, really unsafe technologies on the streets that can cause real damage.

What do you think the media and public often get when trying to understand pilots?

ZIPPER: The media piece is different from the public piece. Let's focus on the public first.

People often don't like change … and I think that transparency with the public about the societal goal that is being tested through a pilot can help people understand why this new technology is on the streets. If a Department of Transportation (DOT) makes clear [their reasons for testing a technology], I think that can build credibility and reduce pushback against the technology. It also frankly builds confidence in government, period, when it’s clear what the goals are of something risky.

It also is a safeguard if the results come back negative. If the hypothesis is not confirmed — if microtransit doesn't save money, or if scooters don't reduce car trips — I think it's terrific if the city can come out and say look, we ran the pilot, we collected the data, our hypothesis didn’t work out the way we thought. So now we’re going to make the appropriate decision.

That's not failure. That's new knowledge created that leads to better policy. And I think the public is more likely to see it that way if there's transparency upfront of what the goals are.

So, the media's role here is to hold public servants and officials accountable. Oftentimes, pilots are launched because local officials want credit for innovation. In my view, the thoughtful members of the media will push right past that and [ask], how is this going to impact Jane Doe's life? Why should taxpayer dollars be spent on this?

My encouragement of the media is to always ask the question with new technologies … Why should somebody on the street actually care about what results from this pilot? How might it improve their life?

When it comes to that transparency, what suggestions do you have for governments to better inform residents?

ZIPPER: It really depends on the technology. I have seen transit agencies running new pilots literally put updated data every couple days on their website, which I think is really impressive when you're able to do that. It does build transparency and I think more confidence in decisions that are ultimately made.

When you think of what happened with scooters, a lot of jurisdictions including Arlington County, VA, did focus groups and interviews with people who were using them to understand not just whether the hypotheses that the county had were being confirmed or rejected, but also if there were other issues coming up that they hadn’t even realized.

That kind of engagement with people who are actually using the new technology I think is great, because it can allow you to collect information beyond the very specific hypotheses that you wanted to test upfront.

What are the obstacles to collecting and using data in mobility pilots? Is that starting to improve at all?

ZIPPER: It’s something that's really important for a city or transit agency to clarify before the pilot launches. You may need to get data and validate that data from the mobility provider itself, so you need to make that clear before the vehicles hit the street.

And then there's the question of, how do you actually have capacity within a transit agency or a city to evaluate the data that’s collected? In my view, one clever thing that can be done is to leverage university capacity to do that. For example, Chicago worked with DePaul University on its scooter pilot analysis. I think that kind of approach, when you have those sorts of partners, can be really useful for city agencies that are often stretched thin.

If I'm a local government and I'm trying to launch an e-scooter pilot, what do you think are the key first steps for me to see success?

ZIPPER: The first question to ask is, why do I want this? What ultimate goals that we have in our comprehensive plan, or that my mayor has established, does this technology align with? Maybe the goal of my city is to reduce car trips. Maybe it's to provide means of transportation for those who live in areas that aren’t served by heavy rail. Once you know what those goals are, that then lets you figure out, what are the statements I hope I can make about e-scooters in my community?

Once you have that, then I think it becomes a data question to say, how do we actually collect the data to be able to verify whether this statement is true or not in a limited geographic area? It can also make sense to actually talk to the providers themselves to say, this is what we want, help us think about how to set up this pilot in a way that allows us to test this. I think getting feedback about the pilot design from the companies that want to participate can be really helpful.

Once you've identified what the questions are that you want to test, it's a matter of saying okay, we know we'll get the data in, now how do we evaluate it? Do we do it internally with our staff capacity or do we need external partners? Who can I call to get to support us?

And then you announce the pilot and say here’s what we’re doing and why. Here's two different ways it may go: we hope to see this, hold us accountable to that as the pilot runs, then ultimately the results are made public and then you base policy decisions on the results that come out.

Any final insights to add?

ZIPPER: The only other point I would make is, pilots are a means of navigating uncertainty. Not every city needs to test the same hypothesis. We would frankly do better collectively if a city that runs a pilot and reaches a conclusion publishes information about that pilot — how the data was collected, what findings were made — then puts it in the public domain. Then, other cities may be able to leverage that information without feeling that they need to go through the same exercise themselves.