Single-Threading in Orleans grains

I am trying to understand single-threading of grains in Microsoft Orleans. I used the code from here and modified it a bit to test my scenarios.

My client code and silo building code

static async Task Main(string args)
{
var siloBuilder = new SiloHostBuilder()
.UseLocalhostClustering()
.UseDashboard(options => { })
.Configure<ClusterOptions>(options =>
{
options.ClusterId = "dev";
options.ServiceId = "Orleans2GettingStarted";
})
.Configure<EndpointOptions>(options =>
options.AdvertisedIPAddress = IPAddress.Loopback)
.ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning).AddConsole());

using (var host = siloBuilder.Build())
{
await host.StartAsync();

var clientBuilder = new ClientBuilder()
.UseLocalhostClustering()
.Configure<ClusterOptions>(options =>
{
options.ClusterId = "dev";
options.ServiceId = "Orleans2GettingStarted";
})
.ConfigureLogging(logging => logging.AddConsole());

using (var client = clientBuilder.Build())
{
await client.Connect();

var random = new Random();
string sky = "blue";

while (sky == "blue") // if run in Ireland, it exits loop immediately
{
Console.WriteLine("Client giving another request");
int grainId = random.Next(0, 500);
double temperature = random.NextDouble() * 40;
var sensor = client.GetGrain<ITemperatureSensorGrain>(grainId);

// Not awaiting this task so that next call to grain
// can be made without waiting for current call to complete
Task t = sensor.SubmitTemperatureAsync((float)temperature);
Thread.Sleep(1000);
}
}
}
}


My grain Interface and actual grain implementation

public interface ITemperatureSensorGrain : IGrainWithIntegerKey
{
Task SubmitTemperatureAsync(float temperature);
}


public class TemperatureSensorGrain : Grain, ITemperatureSensorGrain
{
public async Task SubmitTemperatureAsync(float temperature)
{
long grainId = this.GetPrimaryKeyLong();
Console.WriteLine($"{grainId} received temperature: {temperature}");

await Task.Delay(10000);
// Thread.Sleep(10000);
Console.WriteLine($"{grainId} complete");
// return Task.CompletedTask;
}
}


What I am basically doing is that sending requests to grains every 1 second whereas I am allowing each method invocation inside grain to take at least 10 seconds. Now, according to single-threaded execution of grains and the Orleans Runtime Scheduling described here, I expect that the requests will be queued and the next request will not be taken up by grain unless the current request's method completes. However, the console output doesn't corroborate this. The console output is:

Client giving another request
344 received temperature: 8.162848
Client giving another request
357 received temperature: 10.32219
Client giving another request
26 received temperature: 1.166182
Client giving another request
149 received temperature: 37.74038
Client giving another request
60 received temperature: 26.72013
Client giving another request
218 received temperature: 24.19116
Client giving another request
269 received temperature: 17.1897
Client giving another request
318 received temperature: 8.562404
Client giving another request
372 received temperature: 8.865559
Client giving another request
443 received temperature: 5.254442
Client giving another request
344 complete <-------------- The first request completed here
97 received temperature: 19.24687


This makes it quite clear that the next request is being processed by the grain before the current request completes.

Questions:

So, is this a violation of Orleans single-threaded execution model or am I missing something here?

Also, when I use Thread.sleep(10000) inside the grain instead of Task.Delay(10000), I get the same console output almost apart from an extra warning for every request invocation -
Task [Id=1, Status=RanToCompletion] in WorkGroup [Activation: S127.0.0.1:11111:270246987*grn/6424EE47/00000028@cafcc6a5 #GrainType=Orleans2GettingStarted.TemperatureSensorGrain Placement=RandomPlacement State=Valid] took elapsed time 0:00:10.0019256 for execution, which is longer than 00:00:00.2000000.
Does this mean that every grain should process within 200ms ideally? What happens if the grains process for more time?

Task [Id=1, Status=RanToCompletion] in WorkGroup [Activation: S127.0.0.1:11111:270246987*grn/6424EE47/00000028@cafcc6a5 #GrainType=Orleans2GettingStarted.TemperatureSensorGrain Placement=RandomPlacement State=Valid] took elapsed time 0:00:10.0019256 for execution, which is longer than 00:00:00.2000000

TemperatureSensorGrain(grainId: 344)

TemperatureSensorGrain

1 Answer
1

As @DanWilson says in the comments, you're observing this behavior because each call is being made on a separate grain.

In Orleans, each grain is effectively single-threaded, but not the entire silo or cluster. That means that many grains can execute at the same time and it means that adding more cores to your host or adding more machines will allow you to scale your service.

Modifying your code to select a grainId just once (by moving that outside of the loop), I see this sample execution:

grainId

137 received temperature: 18.74616
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
137 complete
137 received temperature: 20.03226
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
Client giving another request
137 complete
137 received temperature: 21.4471


Which is what you would expect: many requests are being enqueued (one per second), but each request takes 10 seconds to complete before the grain can begin processing the next request.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.