Sitecore Content Search with Azure - Index Out Of Range Exception

Recently, I was working on my first Sitecore implementation that used Azure Search as its search provider. The Sitecore version was 9.0 Update 2. The environment was Azure PaaS, but the developers did not have access to their own Azure Search instances for local development. So, we made the mistake of using a different provider locally (Lucene). Since Sitecore uses totally different libraries for Content Search depending on which search provider you use, this was a bad strategy.

The first indication that a problem was happening was that searches which worked just fine in the Lucene environment were throwing this exception when in the deployed environment with Azure Search.

[ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index]
System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource) +72
Sitecore.ContentSearch.Azure.Query.SearchResultIterator.get_Current() +43
Sitecore.ContentSearch.Azure.Query.<GetEnumerator>d__8.MoveNext() +61
Sitecore.ContentSearch.Azure.Query.<GetSearchHits>d__15.MoveNext() +154
System.Linq.WhereSelectEnumerableIterator`2.MoveNext() +124

There was an exception happening inside the Sitecore.ContentSearch.Azure DLL and we couldn't figure out why. After some time, we finally got access to new Azure Search instances for each of the developers. Then, with the help of a few developers at HI, I got set up with dotPeek, attached the DLL to a symbol server, and started debugging. What we found was that this is the code where the exception is being thrown, from Sitecore.ContentSearch.Azure.Query.SearchResultIterator:

public Dictionary<string, object> Current    {      get      {        try        {          return this.searchResults[this.position];        }        catch (IndexOutOfRangeException ex)        {          throw new IndexOutOfRangeException();        }      }    }

this.position was getting outside the bounds of this.searchResults. Not only this, but the IndexOutOfRangeException was being caught and then a new one was being thrown instead of using the original one with the useful stack trace data.

Why was this exception occurring?

The answer is that our code was enumerating results more than once, and only using the enumerated results the second time. For each time MoveNext() was called on the enumerable, this.position was being incremented, but never got reset. The Sitecore code that connects to Lucene doesn't use this same behavior, and I can only assume Solr would also not have the same issue. After raising this issue with Sitecore Support, they filed a bug ticket about it (214219).

In the meantime, we were stuck with this version of the Sitecore Content Search code, and had to work around the issue by ensuring that for any given request made to Sitecore Content Search, the SearchResults.Hits property only gets enumerated once for the search context, and dispose the search context afterwards so that it can't accidentally get enumerated again.